The "Self-Descriptive" Neo4j Graph Database (Part 2)

Author: Jim Salmons, Project Director - The Softalk Apple Project and the FactMiners Ecosystem

Note: This second-part of my original metamodel subgraph GraphGists gets into some 'niddling detail' related to graph database design issues that may be beyond the level of your interest. For a more reader — and video viewer — friendly version of much of this material, please see the GraphGist Edition of my presentation at the #MCN2014 conference of the Museum Computer Network: Where Facts Live: Exploring the Metamodel Subgraph of a FactMiners' Fact Cloud.

In the first of this two-part GraphGist, I described the basics of the metamodel subgraph design pattern and we looked at a very basic example. With that, we begin here to tackle the interesting use case that inspired my development of the FactMiners ecosystem in the first place.

Softalk Magazine was published monthly in four volumes between September of 1980 and August of 1984. Its 9,300 pages provide an invaluable detailed chronicle, month by month, of the early days of the Microcomputer Revolution. While the FactMiners social-game ecosystem is intended to have a life and reach of its own, the inspiration for FactMiners and the first Fact Cloud to be developed will be the one described in this series and future GraphGists. I and the many Friends of Softalk invite you to learn more about this great magazine and The Softalk Apple Project. And you can follow this link to learn a bit more about my personal 'pay it forward' inspiration to bring the FactMiners ecosystem to life.

Bottom line. Creating the definitive on-line digital archive of Softalk Magazine is a great idea. And 'birthing' the FactMiners social-game ecosystem as a means to explore this archive is going to be both a LOT of fun and full of interesting challenges. So let us jump right in…​

The basic idea behind FactMiners is to harness the player energies and interest in playing the game to create a crowdsource-resource for museums and archives with on-line digital collections interested in creating a Fact Cloud (i.e., a "self-descriptive" graph database) as a supplemental educational and research resource within their on-line collections.

Within the role-based player-world of FactMiners, museum and archive representatives – usually an employee or other authorized agent – are simply FactMiners players with the "Fact Cloud owner/creator" permission enabled on their player account at the http://www.FactMiners.com player community. When a museum or archive is ready to add a Fact Cloud to its collection, the collection’s player will use the game app or a browser to start the Fact Cloud Wizard.

Like wizard-style set-up features you have used before, the Fact Cloud Wizard is a knowledge-based programmed interaction (i.e., a guided 'conversation' UI design-wise) that helps the wizard-user create something complex by breaking it down into simpler small decisions and step-wise configuration activities.

The Cypher snippets we look at in this second part of my GraphGist series are presented in 'conceptual chunks' that would wizard-wise be generated and executed in piecemeal fashion based on incremental guided questions posed by the wizard. So I will not belabor the fine-grained detail of the wizard interaction here. We can simply recognize that the metamodel we create in this second part of this GraphGist series could reasonably be generated by the to-be Fact Cloud Wizard that will be part of the platform developed by the FactMiners.org Open Source developer community. (Just a placeholder page ATM…​ Interested? It is next up 'TODO' after this gist! :-) )

In part 1 of this GraphGist series, I introduced a barebones metamodel subgraph that looked like this first diagram. The META subgraph only needed the Nodes and Relationships subsets to conceptually organize the elements of the introductory example.

Our first step is to rough in the structure of Softalk magazine into the our Fact Cloud metamodel. You will see how we can expand the use of our node-labeling technique to handle the increased complexity of our real world, domain-specific example.

To include the structure of Softalk Magazine in our Fact Cloud metamodel, I adopt a label-naming convention to provide a subset-containment semantic to our metamodel organization. Every node in the metamodel subgraph has two labels:

Note: The current Neo4j implementation treats multiple labels on a node as a collection of independent set membership 'tokens'. I could just as easily use multiple standard Neo4j labels and simply adopt the convention of writing these labels in 'containment-conceptual' order. The CamelCase approach is safer from unanticipated gotchas while also allowing me to explore the use of regexes in Cypher label matching clauses to get most of what I might want in a path-containment semantic without having to wait for it to be added to the Neo4j feature-set. In version 0.1, a `backticked.dotted.path.string\` semantic was used. In this version 0.2, I am adopting the CamelCase convention so the brilliant code wizards at https://structr.com, Axel and Christian, can do structr awesomeness helping to make the FactMiners Fact Cloud Wizard and FactMiners.org website REAL, real soon!!! What we sacrifice in readability, we gain in implementation awesomeness. This is a very good trade. :-) Perhaps the most unfortunate impact of this convention is the need to call PropertyMaps Propertymaps. This is a work in process.

Nodes in the Softalk Fact Cloud metamodel have at least two 'native' properties:

By 'native' properties, I mean properties entered explicitly on the node. As you will see in a later section of this gist, we will use a "divide and conquer" strategy to handle node and relationship properties as Propertymaps that can be associated automatically by node type matches, or manually by way of an attachment-association semantic.

The 48 issues of Softalk Magazine provide a detailed time-capsule snapshot of the earliest days of the microcomputer revolution; September 1980 through August 1984. The 48 issues are clustered in four annual Volumes that make up the full publication run of the Magazine. Each issue consists of a number of Pages.

Here is a quick refresher on the classic structure of a magazine (some of which here are Softalk-specific examples of the kind of non-feature elements found in the front and back of a typical magazine):

We start by creating the metamodel elements that describe the essential Magazine Structure.

For our Softalk Magazine Fact Cloud, our archive-rep player uses the Fact Cloud Wizard to respond to a series of guided prompts about the elementary units and composition of the digital collection to be fact-mined:

In the above Cypher queries, our player used the Fact Cloud Wizard to describe the decomposition of source material structure resulting in Wizard-generation of the structural nodes and PART_OF relationship nodes specific to the SourceStructure subset of elements in the metamodel. Of course, no players – even those creating and maintaining the Fact Cloud metamodel – have to know or understand metamodels, their interpretation, or anything. It is the Wizard that writes the code we are exploring here.

Depending on how your browser or device has rendered the graph visualization of these new nodes, it may not be apparent that the wizard has created a snake-like path of alternating structural element nodes and PART_OF relationship nodes that model the PART_OF relationship in the actual database. If the Wizard discovers later in the configuration interaction that the PART_OF relationship is contextually complex, the _metamodel node about that relationship serves as the 'root' of the subgraph in the metamodel that will describe that relationship in the non-meta database.

Continuing on in ELIZA-like conversation with the Fact Cloud Wizard, our Fact Cloud Creator player continues drilling down on the structure of the collection items to be fact-mined. Our player next tells the Wizard about all the various magazine structural elements that could be found on any given page – magazines are a complex document structure when looked at fact-mining-wise. Here are a few of the most obvious of these "page parts" that the Wizard would add to further characterize the Softalk Magazine archive within the Fact Cloud metamodel:

The rather lively and unpredictable rendering behavior of the GraphGist "bouncy-ball" graph visualization can make this emerging Magazine source collection structure a bit hard to see. But if you reach in and yank the nodes around as necessary, you will see a graph-form I call 'the sperm' wherein the 'tail' is the PART_OF path that deconstructs the Magazine down through to the constellation (the sperm 'head') of page-parts surrounding the Page metamodel node.

In the next section, our player will use the Wizard to turn this sperm into a dumbbell.

At the most basic level, Softalk content is about People, Products, Companies, Technologies, Locations, and Events – among many other things. But we will keep this GraphGist focused on the bare essentials needed for our "fact discovery and entry" gameplay use case scenario in the third part of this GraphGist series.

The following Cypher query creates the metamodel nodes that represent five types of editorial Content found in Softalk Magazine:

These Content elements are used as the 'thing' nodes 'related' in a FactMiners Fact Cloud 'fact' which is – at its simplest – the elementary semantic expression of the graph database, i.e., (thing_A) - :IS_RELATED → (thing_B).

As we evolve the metamodel for the Softalk Magazine Fact Cloud, you can start to see some of the structured ways that these Content elements can be related to specific fine-grained elements of the Magazine structure. For example, in the third part of this GraphGist we will walk through a 'fact discovery and entry' gameplay scenario about entering information found in the famous Softalk Top 30 bestseller lists. After we take a quick look at how the metamodel handles node and relationship properties, we wrap up this part of series by finishing off the metamodel coverage of the Bestseller list document structure.

Property metamodeling will be important in the FactMiners design. This is most likely where we will want to keep 'hint pack' run-time information so that metamodel-aware thin-client applications – e.g., the FactMiners social-game app – can get fine-grained helpful information to perform compliant data access, editing, and visualization tasks.

Although I can think of a few different ways to metamodel properties, two considerations have led me to an initial approach:

Knowing these points, and to keep from having exploding visual complexity as more and more metamodel elements have more and more interconnects, I have adopted the following mechanism to provide basic property modeling in the Softalk Fact Cloud metamodel:

The anticipated use of property storage for 'widget-level hints' to dynamically configure editing and visualization tasks is reason enough to assume that this whole area of metamodeling of properties will be its own interesting and challenging domain. It is also reasonable to assume 'separation of concern' to help simplify our Fact Cloud Wizard development requirement. So treating Propertymaps this way may be useful longer-term as we develop the Property-Builder component of the Fact Cloud Wizard.

Here is a sample of how metamodeling Propertymaps works to provide some basic properties to the PERSON and COMPANY SourceContentNodes. We first create the 'root' node in the SourceContentPropertymaps subset of the metamodel. We then create and link a Propertymap node for any type of SourceContentNodes elements that need to model properties of data in the Neo4j database. And finally, we create example property nodes and organize them in the SourceContentNodesPropertymaps subset (as this is the 'workspace' of the Property-Builder of the Fact Cloud Wizard):

That is about as much as I need to cover about properties in order to support the gameplay interactions of the use case scenario in the third part of this GraphGist series.

Unfortunately, we are also starting to get to the point where the current state of control over GraphGist graph visualization gets a bit unwieldy going forward. You start to see 'islands' of Propertymap clusters as metamodel elements need to describe properties of nodes and relationships in the non-meta data. But, as before, if you 'reach in and yank things around' you can get a pretty good idea of how our Softalk Fact Cloud metamodel is coming along.

Of course within the controlled user environment of the Fact Cloud Wizard, good UI design will make metamodel creation, maintenance, and extension, if not fun, at least organized and doable. In the meantime, it is kind of fun to grab a node and yank it around a bit.

So far, we have carved up the Pages of the Magazine into (currently rectangular) areas which map to the document-structuring elements like masthead, list of advertisers, column, feature, review, bestseller lists, advertisements, etc. Each of these page-part regions may have structure as well. A very data-rich example of such a subpart-structured element of Softalk Magazine is its various bestseller lists. Each month a Top 30 list rated Apple computer software popularity regardless of category. Top 10 lists rated business applications, games, and educational software, etc. as the industry and consumer marketplace evolved. These lists were rigorously and independently researched and reported by Softalk and are, therefore, among the interesting and historically valuable data locked within the Softalk Magazine archive Pages.

Each bestseller list has its prescribed number of list item entries. These bestseller listing item nodes become a new leaf in the "PART_OF" path of elements that model the structure of the magazine. I have modeled three example lists here. As they each have the same line item structure, these three page-parts each are end-nodes of newly added PART_OF relationships that link a bestseller list item into the Magazine structure:

When we examine a bestseller list line item listing, we can begin to see how the "fabric" (or "weave") of the "facts" in the Softalk Fact Cloud will capture information that is "tied to" (i.e., has meaning in the context of) the structure of the Magazine – the ordinal position of the item listing within the bestseller list as well as the monthly statistical index computed by the list researchers – and those that relate the listing to "domain objects" in the Real World that the Magazine covers in its editorial – Persons are Developers of (software) Products and Companies are Publishers of (software) Products. As we are evolving this metamodel in a piecemeal exploration, we can add this content/structural mapping to the metamodel.

As the close-up in the collage image of "full of facts" shows, bestseller listing items have (ordinal) position and (monthly rating) index properties. Since our Wizard has not yet created any propertyMaps in the *SourceStructureRelationshipsPropertymaps subset, it first creates the 'root' where the Property-Builder component of the Wizard maintains its contributions to the metamodel, then creates the required Propertymap elements:

Now we get to a place where things get interesting…​ We will create our first metamodel construct that relates something from the magazine editorial content (i.e., a SourceContentNode node) to a magazine structural element (SourceStructureNode).

Top X list items are associated with a software:PRODUCT. The item listing includes the name of the developer and company that published the product. The listing, therefore, "reaches through" the ON_LIST relationship to confirm or create relations between the product and its developer:PERSON and publisher:COMPANY, so we create appropriate relationships:

I am pretty sure that you will require "serial node yanking" to zero in on these final interesting additions that we just made to the metamodel. The "content meets structure" aspect of these last additions is clearly evident in the ON_LIST relationship between the bestseller list item structural element and the Product content element. But it is really the "reach through" aspect that is most interesting and best demonstrates the power of a "self-descriptive" graph database. And that is, I believe the metamodel subgraph design pattern will be especially useful for applications with loosely-coupled but richly-semantic data.

Yes, it is interesting that we have drilled down to the point to see a fine-grained example of how Magazine content and structure intertwine in a structured sense – e.g., list of advertisers, buying guide directory, etc. But the bigger challenge in fact-mining a magazine is that there are so many different ways that any given "fact" can be manifest. In the final Cypher snippet we created the DEVELOPER and PUBLISHER relationships as a result of finding that relationship expressed in a bestseller list line item. But taking the example from the "full of facts" image here, there will be MANY places in Softalk magazine where we find "sightings" (instances) of the facts that "Dan Gorlin is the developer of Choplifter." and "Choplifter was published by Broderbund Software." Indeed, the number and distribution of these fact instances is interesting new data that will be generated as a product of creating the Fact Cloud.

One thing that really excites me about this project is what I hope to see when we fully fact-mine the 48 monthly issues of Softalk magazine. When all the "facts" in this remarkable and historic magazine are in a "self-descriptive" Neo4j graph database, I believe we will have a fascinating and valuable new "lens" on a most remarkable period in the History of Technology.

If either the means – designing and developing the FactMiners ecosystem – or the ends – helping to create the Softalk Magazine Fact Cloud – interest you, please do not hesitate to contact me. (Twitter: @Jim_Salmons)

Thank you for sticking with me through the first two parts of this GraphGist series.

Let’s move on to part 3 of this GraphGist series where we will explore a 2-player gameplay scenario that will exercise the metamodel we just built here. (Actually, it will be a while before part 3 is available as it is currently in early draft, and part 4 is in the queue behind that. But first…​ I’m going to get the FactMiners.org site going…​)