Riding the Dromedary

Before I really get started on this entry there is something I have to get out of my system. There are no camels in Egypt (or most of Africa for that matter). There are dromedaries (camelus dromedarius) in Egypt. The animal with two humps we've come to associate with the word camel (camelus bactrianus) is dominant in Asia (and Hollywood) only. The ship of the Egyptian desert is a dromedary.

Right, a couple of weeks ago the NetKernel newsletter contained a festive Camel recipe. And all banter put aside it looked like an interesting thing to study. At first though - forgive me Peter - I couldn't make heads or tails out of it. Only bumps.

So I set aside a whole day to figure it out and to create my own example. Which I'll explain here.

First we need some definitions. Here is what Apache Camel is :

A powerful open source integration framework based on known Enterprise Integration Patterns with powerful Bean Integration.

Isn't that wonderful? Actually it is about (message) routing and transports. I'm going to assume for a moment that the routing bit (I'll come back to it in the example) is clear and explain about transports. In NetKernel a transport typically sits on the border. The border between NetKernel and the outside. What it does? It mediates. Brings stuff inside the ROC abstraction and takes stuff out. Lets call this stuff messages.

We all know a couple of transports. The FrontEnd HTTPFulcrum and Backend HTTPFulcrum are the prime examples. These are HTTP transports able to take your message (a HTTP request), bring it into the ROC abstraction where we can process it and take a message back (a HTTP response).

It is easy to overlook (I know I did when I started with NetKernel) that there are many other possible transports. Have a look at the Apache Camel Components list for just a small selection (bookmark that page while you are at it). Yes, a small selection, for you can obviously create your own protocols and some of the listed implementations cover more than one protocol.

Another important word. Protocol. Let us give that a definition (from Wikipedia) too :

A communications protocol is a system of digital message formats and rules for exchanging those messages in or between computing systems and in telecommunications.

Translated ... a given transport needs to know what the messages (have to) look like in other to do its work. Sounds logical.

Ok, with the definitions out of the way it is time for some action. My example creates a simple transport that more or less implements the Echo Protocol. Now, I do know how to implement that protocol and if you want to learn more about that (and other network protocols), I advise reading the Unix Network Programming books by Richard Stevens. However, the whole point of Apache Camel is that you do not have to bother with the gory details. For general network protocols you can use the Apache Mina component.

Here is the relevant part of module.xml :
  <rootspace
    name="Echo transport"
    public="true"
    uri="urn:org:practicalnetkernel:tpt:mina:echo">
    
    <prototype>
      <class>org:practicalnetkernel.tpt.mina.endpoint.EchoTransport</class>
      <id>camel.EchoTransport</id>
    </prototype>
    <endpoint>
      <prototype>camel.EchoTransport</prototype>
    </endpoint>
  </rootspace>

You might wonder why we work with a prototype. Well, the answer is given by the existing HTTP Transports. Wouldn't you agree that the Frontend and Backend HTTPFulcrums are actually two instances (with slightly different configurations) of the same HTTP Transport? They are. And the same could apply for all transports. Hence the use of a prototype.

You are going to need libraries. My module's lib directory :
camel-context-2.9.0.jar  -> from Apache Camel download
camel-core-2.9.0.jar     -> from Apache Camel download
camel-mina-2.9.0.jar     -> from Apache Camel download
mina-core-1.1.7.jar      -> from Apache Mina download
slf4j-api-1.6.1.jar      -> from Apache Camel download
slf4j-log4j12-1.6.1.jar  -> from Apache Camel download
log4j-1.2.16.jar         -> from Apache Camel download

Note that there is a more recent version of Apache Mina available. The Apache Camel component (camel-mina) does not know of it yet however.
Remember the components list you had to bookmark ? That list will indicate which third party libraries you need for a given component.

The transport class looks like this :

package org.practicalnetkernel.tpt.mina.endpoint;

import org.apache.camel.CamelContext;

import org.apache.camel.builder.RouteBuilder;

import org.apache.camel.impl.DefaultCamelContext;

import org.apache.camel.impl.SimpleRegistry;

import org.netkernel.layer0.nkf.INKFRequestContext;

import org.netkernel.module.standard.endpoint.StandardTransportImpl;

public class EchoTransport extends StandardTransportImpl {

  public static final String ECHONKTRANSPORT="ECHONKTRANSPORT";

  private CamelContext mCamelContext;  

  protected void postCommission(INKFRequestContext aContext) throws Exception {

    SimpleRegistry lRegistry = new SimpleRegistry();

    lRegistry.put(ECHONKTRANSPORT, this);

    

    mCamelContext = new DefaultCamelContext(lRegistry);

    mCamelContext.addRoutes(

      new RouteBuilder() {

        public void configure() {

          from("mina:udp://0.0.0.0:8181?sync=true")

          .to("class:org.practicalnetkernel.tpt.mina.endpoint.EchoHandler?method=onMessage");

        }

      }

    );

    mCamelContext.start();

  }

  

  protected void preDecommission(INKFRequestContext aContext) throws Exception {

    mCamelContext.stop();

  }

}

The important part here is the route (told you we would discuss that) :

from("mina:udp://0.0.0.0:8181?sync=true")

          .to("class:org.practicalnetkernel.tpt.mina.endpoint.EchoHandler?method=onMessage")

Here lies the real strength of Apache Camel. Without knowing anything about Apache Mina you can figure out that something (our transport) will be listening at port 8181 (normally you'd make that a parameter) and pass the messages to the onMessage method of an EchoHandler class. The sync=true option indicates that the client will wait for a response. You can find all that information on the Apache Camel Mina page.

The handler looks like this :

package org.practicalnetkernel.tpt.mina.endpoint;

import org.netkernel.layer0.nkf.INKFLocale;

import org.netkernel.layer0.nkf.INKFRequestContext;

import org.apache.camel.Message;

public class EchoHandler {

  public void onMessage(Message aMessage) throws Exception {
    // Determine the transport so we can get the NetKernel context.

    EchoTransport transport=(EchoTransport)aMessage.getExchange().getContext().getRegistry().lookup(EchoTransport.ECHONKTRANSPORT);

    INKFRequestContext aContext=transport.getTransportContext();

    // Getting some extra information from the message.
    Object origin = aMessage.getHeader("CamelMinaRemoteAddress");

    // NetKernel processing. Very limited in this case.

    aContext.logRaw(INKFLocale.LEVEL_WARNING, origin.toString() + " - " + aMessage.toString());

    // Setting the transport response

    aMessage.getExchange().getOut().setBody(origin.toString() + " - " + aMessage.toString());

  }

}

And that is it. The incoming message is extended with the address of the caller, send as a warning to the NetKernel log (to prove we came inside the ROC abstraction) and returned to sender.

The original example did exactly the same for HL7, but do you know how to send an HL7 message (to test the transport)? Good on you, I do not. For the people that have the same issue with the above, here is a small  standalone Java program :

import java.net.* ;

/**

 *  A simple datagram client

 *  Shows how to send and receive UDP packets in Java

 *

 *  @author  P. Tellenbach,  http://www.heimetli.ch

 *  @version V1.00

 */

public class DatagramClient {

  private final static int PACKETSIZE = 100 ;

  

  public static void main( String args[] ) {

    // Check the arguments

    if( args.length != 2 ) {

      System.out.println( "usage: java DatagramClient host port" ) ;

      return ;

    }

    DatagramSocket socket = null ;

    try {

      // Convert the arguments first, to ensure that they are valid

      InetAddress host = InetAddress.getByName( args[0] ) ;

      int port         = Integer.parseInt( args[1] ) ;

      // Construct the socket

      socket = new DatagramSocket() ;

      // Construct the datagram packet

      byte [] data = "Hello Server".getBytes() ;

      DatagramPacket packet = new DatagramPacket( data, data.length, host, port ) ;

      // Send it

      socket.send( packet ) ;

      // Set a receive timeout, 2000 milliseconds

      socket.setSoTimeout( 2000 ) ;

      // Prepare the packet for receive

      packet.setData( new byte[PACKETSIZE] ) ;

      // Wait for a response from the server

      socket.receive( packet ) ;

      // Print the response

      System.out.println( new String(packet.getData()) ) ;

    }

    catch( Exception e ) {

      System.out.println( e ) ;

    }

    finally {

      if( socket != null )

        socket.close() ;

    }

  }

}


Use that and see the transport at work. Congratulations, you've just created a new NetKernel transport!


Your challenge for this week is to create your own NetKernel transport with this Apache Camel route :
  from("timer://foo?fixedRate=true&delay=0&period=60000")

  .to("http4://api.twitter.com/1/statuses/public_timeline.xml")

  .to("xslt:resources/stylesheets/gullet.xsl")

  .to("class:org.practicalnetkernel.tpt.tweet.endpoint.PublicTweaterStomach?method=onMessage")
Can you get it to work? What does it do? Send in your solutions to practical<dot>netkernel<at>gmail<dot>com and I'll discuss them in a future post.


While the challenge transport is fun to make (do it, you'll see!), it is also an example of convenience overkill. Once you get used to Apache Camel, you want to ride it all the time. And use it for things that you should do inside the ROC abstraction (where you benefit from all that has to offer). So as always ... use it, do not abuse it. Camels don't actually spit, they throw up and you don't want to be caught in the vomit ...


One last remark to finish this long-winded entry. The animal on the Apache Camel main page ... is a dromedary!

front page

There is no doubt that you know where the following image comes from:

It has a somewhat cryptic message : If required your dynamic import application space can mask the root path. Ok. Fine. Thumbs up. But ... did you get that?

What it means is that you can replace the default root page with your own. Cool. But how?

Now I'm a bit lost when I have to create a front end (my graphical skills are under par ... working on it :-), so I get help. The Skeleton Boilerplate is excellent (thanks go to Joe Devon for pointing it out).

Serving it in NetKernel is easy. Dump it in the resources directory of a new module, customize the webpages as you like and serve it up with the webserver pattern :

  <rootspace>    

    <fileset>

      <regex>res:/etc/system/SimpleDynamicImportHook.xml</regex>

    </fileset>

    <fileset>

      <regex>res:/skeleton/(.*)</regex>

      <rewrite>res:/resources/$1</rewrite>

    </fileset> 

  </rootspace>

However, that leaves one problem. You can now see your pages by requesting http://localhost:8080/skeleton/index.html, but http://localhost:8080/ is still showing the default page. Wasn't it our purpose to change that?

Yes indeed and there are several ways to accomplish it. Here is one:

  <rootspace>    

    <fileset>

      <regex>res:/etc/system/SimpleDynamicImportHook.xml</regex>

    </fileset>
    <mapper>

      <config>

        <endpoint>

          <grammar>

            <simple>res:/</simple>

          </grammar>

          <request>

            <identifier>active:redirectHTML</identifier>

            <argument name="url">/skeleton/index.html</argument>

          </request>

        </endpoint>

      </config>

      <space>

        <import>

          <uri>urn:org:netkernel:web:core</uri>

        </import>

      </space>

    </mapper>

    <fileset>

      <regex>res:/skeleton/(.*)</regex>

      <rewrite>res:/resources/$1</rewrite>

    </fileset> 

  </rootspace>

As you can see we redirect the root request to the index.html page of the Skeleton Boilerplate. The webserver pattern takes over from there.

This might not be the most elegant solution, but it works. If you have a better solution, step forward and send it to me at practical<dot>netkernel<at>gmail<dot>com. The rules are simple, I have to be able to dump the Skeleton Boilerplate in a module and without any change to the boilerplate itself (no, not even to index.html) it gets served and the index.html file becomes the root page of the NetKernel instance (http://localhost:8080/).

The price? I'll discuss your solution (giving you full credits) in a next blog entry. Gentlemen and Ladies, faites vos jeux!

Accessible Accessors

Now, say the title out loud quickly five times in a row and I'm sure you'll qualify for House Slytherin ...

In a previous entry I sort of hinted at the fact that in order to write core tools for NetKernel, you need Java. Since NetKernel is developed in Java I stand by that statement.

However, it is not impossible to use one of the dynamic languages and if you are addicted to one of those you can (and should) develop at least some of your tools in that language.

Which brings us to the workhorses of NetKernel, accessors. Check your local documentation for the explanation, In this entry I want to focus on the Service Accessors. Those accessors are resource transformation functions or in simpler words : they compute.

The fact you'll notice quickly about accessors is this : they require a class. Is that then the end of this entry ? No, for there is an alternative :

    <accessor>
      <id>python.simpleaccessor</id>
      <verbs>SOURCE,EXISTS</verbs>
      <prototype>PythonPrototype</prototype>
      <grammar>
        <active>
          <identifier>active:simpleaccessor</identifier>
          <argument name="foo" min="1" max="1" desc="foo"/>
        </active>
      </grammar>
      <script>res:/resources/python/simpleaccessor.py</script>
      <name>Python Simple Accessor</name>
      <description>python simple accessor</description>
    </accessor>

Instead of a class you can also pass a prototype (which is a class that provides you with an instance of the required runtime).

Indeed, I use Python. So shoot me. Anyway, here is the program.

from org.netkernel.layer1.representation import IDeterminateStringRepresentation;
from org.netkernel.layer0.nkf import INKFRequestReadOnly;
from org.netkernel.layer0.nkf.impl import NKFResponseImpl;


def onSource(lFoo):
  return context.createResponseFrom(lFoo + " was SOURCED");


def onExists(lFoo):
  return context.createResponseFrom(lFoo + " does EXIST");


def onError():
  return context.createResponseFrom("Error");


aFoo         = context.source("arg:foo", IDeterminateStringRepresentation).getString();
lThisRequest = context.getThisRequest();


requesthandler={
  INKFRequestReadOnly.VERB_SOURCE: onSource,
  INKFRequestReadOnly.VERB_EXISTS: onExists}

response = requesthandler.get(lThisRequest.getVerb(), onError)(aFoo);

If you've looked at accessors implemented in Java, you'll notice that those always implement a class that extends the StandardAccessorImpl class. Now, Python does have classes and at first I tried that (and it didn't work), until I realised that I was too late. The prototype is the class, once in the script you are already inside the accessor.

To show what my simple accessor can do (not a lot), here are two screenshots from executions in the request resolution trace tool (click to enlarge).

This entry is maybe (slightly) above the level of my intended audience. My main point is that your investment in a scripting language is not wasted. You'll automatically drift to Java as your tools become more and more core, but scripting languages definitely do the job.  

Give me the tools ...

Succesful Resource Oriented is about composing small tools into a potent whole. Which sounds a bit like witchcraft, but it is not (well, most of the time it is not :-). It is however important to know the available tools. Here's something I cooked up in nCoDE not too long ago (click to image to get a decent look) :

Did you know about the netkernel:/ address space ? I did not, until I started wondering how the tools on the Backend HTTPFulcrum were created. Several of them are about modules and it is not like the 1060 Research people to do the same thing twice.

Only afterwards I discovered the documentation.

There are two lessons here :
1) Know the available tools. Search the documentation and if you can not find what you are looking for ... there is no shame in asking around.
2) When writing new tools (you will, it is inevitable), remember that our kind learns best through mimicry. Look at the available examples and learn from them.

For completeness, here are the xsl files from the example.
module.xml
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:nk="http://netkernel.org" 
                version="1.0">
  <xsl:output method="xml" 
              indent="yes" 
              encoding="UTF-8" 
              omit-xml-declaration="yes"/>
  <xsl:param name="moduleid" nk:class="java.lang.String" />


  <xsl:template match="/modules">
    <resultset>
      <xsl:copy-of select="module[id=$moduleid]"/>
    </resultset>
  </xsl:template>
</xsl:stylesheet>

strip_namespaces.xsl
<xsl:stylesheet
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  version="1.0">


  <xsl:output method="xml" 
              indent="yes" 
              encoding="UTF-8" 
              omit-xml-declaration="yes"
              media-type="application/xml"/>


  <xsl:template match="*">
    <xsl:element name="{local-name()}">
      <xsl:apply-templates select="@* | node()"/>
    </xsl:element>
  </xsl:template>
  
  <xsl:template match="@* | text() | comment() | processing-instruction()">
    <xsl:copy/>
  </xsl:template>
</xsl:stylesheet>

As to what my example does ... well ... you'll have to try it yourself, won't you ? Btw, the second XSLT in the example is not really necessary, nor is the XMLTidy step. I just like my results to be human-readable as well.