collusion - a better client

Come one, come all,
Hurry hurry here's your chance,
See the mystery and romance 

Ok, no romance in this episode but quite a bit of mystery. These are exciting times. I recently got pinged by the new CEO of 1060 Research. Yes I know, I did a double-take too. More than one in fact. Peter Rodgers is now VP of Leisure. One Charles Radclyffe (there's a proper Brit name if I ever saw one) has now taken it upon him to try and bring Resource Oriented Computing to the masses.

https://www.deviantart.com/art/At-Library-707812195

A short search on the internet put my mind at ease. Just look at the available pictures for the two gentlemen and combine that with the fact that the 5th Earl of Derwentwater died in 1746. Your penny has not dropped yet you say ? In that case I advise you check out What Happened To Monday. Willem Dafoe is excellent as per usual but that's not the point. The point is ... Yikes Shaggy ... indeed ... and there's five more !

I'm also still working for Neo4j. A year has gone by and what a ride it has been so far. Literally too, I've travelled more distance in the last year than I have in the 44 years before it. Combined.

Today I want to revisit the Neo4j client for NetKernel that I had already created last time. The 1.0.0 version works but has a serious flaw. It creates (and closes) a connection pool for every query that you do. That's not efficient.

This is however not a trivial problem : 

• You've got to keep track and manage a connection pool for each database configuration that you have.
• When a database request comes in you've got to identify the correct connection pool and get a connection/session from that one (and not another). 
• When a configuration changes you've got to close and reopen the connection pool with the new parameters. 
• When a connection pool isn't used you might want to close it to conserve resources.
• ...

Resource Oriented Computing comes to the rescue with two of its tenets.

The first is transreption. Remember the definition for that ? An isomorphic (lossless) transformation of one representation to another. We all understand (if not that definition) the fact that an XML document can be represented as a JSON document without any data loss. Here we take that one step further and state that the configuration for a specific database can be transrepted to the connection pool for that specific database. This solves a lot of the above problems.

The second is the caching mechanism. And there is a reason folks that this is the patented part of NetKernel. In this case it solves all of the other problems. Lets go through a typical scenario :

1. I do a first database request and pass the configuration of the database 
2. A transreption happens (connection pool can not be found in cache) and the connection pool is created 
3. I get a connection/session from the connection pool and my request is executed 
4. I close the connection/session
    
5. I do a second database request and pass the same configuration 
6. The transreption does not happen, the connection pool for that configuration comes from cache 
7. I get a connection/session from the connection pool and my request is executed 
8. I close the connection/session
    
9. I do a third database request and pass a different configuration 
10. A transreption happens (connection pool can not be found in cache) and the connection pool is created 
11. Because memory is - maybe - low the caching mechanism tries to find unused resources in cache and finds the first connection pool. It is closed. 
12. I get a connection/session from the connection pool and my request is executed
13. I close the connection/session
     
14. If unused for a while the caching mechanism may find the second connection pool to be unused and closes it.

This is a very powerful pattern. You find a - slightly more robust - version of it in the urn.org.netkernel.mod.db module and now as well in my urn.com.ebc.neo4j.client-2.0.0 module which you can find in my NetKernel github repository.

Lets just have a look at what the ConnectionPoolRepresentation looks like :
package com.ebc.neo4j.client;

import org.neo4j.driver.v1.*;

public class ConnectionPoolRepresentation implements AutoCloseable {
    Driver mDriver;
   
    public ConnectionPoolRepresentation(String aURL, String aUsername, String aPassword)  {
        mDriver = GraphDatabase.driver(aURL,
                AuthTokens.basic(
                        aUsername,
                        aPassword
                    )
                );
    }
   
    public Session getSession() {
        return mDriver.session();
    }

    @Override
    public void close() throws Exception {
        mDriver.close();
    }

}

Yes I know, there's no error handling yet. But look at the - lack of - size on that thing ! And here's the relevant pieces of code in the actual database-request-handling accessor. The first piece sources the configuration and does the transreption (if necessary) :


        ConnectionPoolRepresentation vCPR = null;
        if (aContext.getThisRequest().argumentExists("databaseconfiguration")) {// databaseconfiguration is an optional argument
            try {
                vCPR = aContext.source("arg:databaseconfiguration", ConnectionPoolRepresentation.class);
            }
            catch (Exception e) {
                throw new Exception("RowsAccessor: no valid - databaseconfiguration - argument");
            }           
        }
        else {// neo4j:databaseconfiguration is the default
            try {
                vCPR = aContext.source("neo4j:databaseconfiguration", ConnectionPoolRepresentation.class);
            }
            catch (Exception e) {
                throw new Exception("RowsAccessor: no valid - neo4j:databaseconfiguration - resource");
            }                   
        }

The second piece is the actual usage of the connection pool :

        // processing
        IHDSMutator vMutator = HDSFactory.newDocument();
        vMutator.pushNode("results");
       
        Session vSession = null;
       
        try {
            vSession = vCPR.getSession();
           
            StatementResult vResult = vSession.run(aCypher);
           
            while (vResult.hasNext()) {
                Record vRecord = vResult.next();
               
                vMutator.pushNode("row");
               
                for (Pair <String, Value> vListEntry : vRecord.fields()) {
                    process_listentry(vSession.typeSystem(), vMutator, vListEntry.key(), vListEntry.value());
                }
               
                vMutator.popNode(); // row
            }           
           
        }
        catch (Exception e) {
           
        }
        finally {
            if (vSession != null) {
                vSession.close();
            }
        }
        vMutator.popNode(); // results
       
        IHDSDocument vRepresentation = vMutator.toDocument(false);
        //

Now, the urn.com.ebc.neo4j.client-2.0.0 module is obviously an utility module so there's no fancy things to show, but these are some of the requests you can do in the request trace tool :

<request>
    <identifier>active:rows</identifier>
    <argument name="databaseconfiguration">
        <literal type="hds">
            <config>
                <url>bolt://127.0.0.1</url>
                <username>neo4j</username>
                <password>xxxYYYzzz</password>
            </config>
        </literal>
    </argument>
    <argument name="expiry">
        <literal type="long">30000</literal>
    </argument>
    <argument name="cypher">
        <literal type="string">MATCH (n) RETURN n LIMIT 5</literal>
    </argument>
</request>

<request>
    <identifier>active:rows</identifier>
    <argument name="expiry">
        <literal type="long">30000</literal>
    </argument>
    <argument name="cypher">
        <literal type="string">MATCH (n) RETURN n LIMIT 5</literal>
    </argument>
</request>

Obviously these requests expect you to have a Neo4j instance running. The second request will create the configuration from the environment parameters of the user running NetKernel. For example on Windows :
set neo4j_databaseurl=bolt://127.0.0.1
set neo4j_databaseuser=neo4j
set neo4j_databasepassword=xxxYYYzzz

An that concludes this post. Next time we're going to take a look at the NeoSemantics work of my colleague and friend Jesus Barrasa and how we can take that to the next level with NetKernel.