firsts

Today is history.
Today there is a NetKernel™ book.

There will be many like it, probably better ones, but this one is first and this one is mine. You can check it out (and order it) on the O'Reilly site. The American and European Amazon sites will follow suit in a couple of days.

However excited (or not) you may be about this, I do urge my fellow Europeans to wait for it to appear on European sites. Given the price you'll otherwise pay it three times over (transport and import taxes) ... and of course I'd rather have you buy three copies instead.

They (never found out who they are) say that pride goes before fall and that they are proportional. I figure I'll end up close to the earth's core then.

Seriously, it is a beginner's book, written from the point of view of my own meanderings through ROC™ and NetKernel™. There's many things I would do/do differently today but at the time it made sense and it will make sense to you too.

Is this first then the end of the free community version ? No ! In fact, now that it is published I'm able to do a - long overdue - refresh of the community version. And work towards a 2nd edition of course.


On this day of firsts, here is a screenshot of something I've been working on recently :

We need to have more fun with NetKernel™ and what better way to have fun than a game ? The first nkMUD (unless you beat me to the release of course ... challengers are always welcome) is in the pipeline, watch this space for updates !

Phew, it's hot here at the earth's core. Enjoy your week while I climb back up !

the nature of ducks

There is a saying that goes like this ... if it looks like a duck, walks like a duck and quacks like a duck, chances are good that it is a duck. That does of course not keep the animal from being 10 feet tall, but even then it probably is a duck.

Here is a table I want to share with you : 

	REST method	ROC verb	Database dml/ddl
create	POST	NEW	INSERT
read	GET	SOURCE	SELECT
update	PUT	SINK	UPDATE
delete	DELETE	DELETE	DELETE

You've probably seen that table before or noticed the similarity, the point I'm trying to make in this post is that ... if it looks like a crud, walks like a crud and quacks like a crud, chances are good that it is a crud.

In the REST world, it is considered bad form to use a GET when you should be using another method. Example
GET http://yourserver/kernelproperty/get/x
GET http://yourserver/kernelproperty/delete/x
is bad and
GET http://yourserver/kernelproperty/x
DELETE http://yourserver/kernelproperty/x
is good.

Yet, from the point of view of most browsers, GET is the only thing you will ever need. And developers follow that adagio in their web applications. Possibly wrong, but hey, it works ...

In the ROC world, all you need is SOURCE, the abstraction requires no more. However, from a code-convenience point of view things look different and the other verbs can be put to good use in creating a model that gives us a handle on the abstraction.

A good place to start is where REST and ROC meet. Now, HTTP is only one entry into ROC, but no method to verb translation is done. Everything comes in as a SOURCE request, you do have access to the httpRequest:/ space though and a very useful resource in there is httpRequest:/method.

An interesting and recent development in NetKernel is the RESTOverlay. I played a bit with that, mixed it with the above thoughts and came up with this :

    <rootspace
    name="System Admin"
    public="true"
    uri="urn:org:elbeesee:ext:system:admin">
    
    <fileset>
      <regex>res:/etc/system/SimpleDynamicImportHook.xml</regex>
    </fileset>


    <overlay>
      <prototype>RESTOverlay</prototype>
      <config>
        <basepath>/elbeesee/</basepath>
      </config>
      <space>
        <endpoint>
          <meta>
            <rest>
              <simple>{accessorname}/{propertyname}/{propertyvalue}</simple>
              <method>PUT,POST</method>
            </rest>
          </meta>
          <grammar>
            <active>
              <identifier>active:restMethodToVerb_pp</identifier>
              <argument name="accessorname"/>
              <varargs/>
            </active>
          </grammar>
          <class>org.elbeesee.ext.system.RESTMethodToVerbAccessor</class>
        </endpoint>
        <endpoint>
          <meta>
            <rest>
              <simple>{accessorname}/{propertyname}</simple>
              <method>GET,DELETE</method>
            </rest>
          </meta>
          <grammar>
            <active>
              <identifier>active:restMethodToVerb_gd</identifier>
              <argument name="accessorname"/>
              <varargs/>
            </active>
          </grammar>
          <class>org.elbeesee.ext.system.RESTMethodToVerbAccessor</class>
        </endpoint>
        <import>
          <private/>
          <uri>urn:org:elbeesee:ext:system:accessors</uri>
        </import>
      </space>
    </overlay>


    <import>
      <private/>
      <uri>urn:org:netkernel:tpt:http</uri>
    </import>
    
  </rootspace> 


And the onSource method (everything that comes in over HTTP is a SOURCE) of the org.elbeesee.ext.system.RESTMethodToVerbAccessor class looks like this :

   public void onSource(INKFRequestContext aContext) throws Exception {
    INKFRequestReadOnly lThisRequest = aContext.getThisRequest();


    // One mandatory argument
    String aAccessorName = aContext.getThisRequest().getArgumentValue("accessorname");
    String aHTTPMethod   = (String) aContext.source("httpRequest:/method");
    
    aContext.logRaw(INKFLocale.LEVEL_DEBUG,"SOURCE HTTPMethod = " + aHTTPMethod);
    
    INKFRequest subrequest = aContext.createRequest("active:" + aAccessorName);
    for (int i = 0; i < lThisRequest.getArgumentCount(); i++) {
      aContext.logRaw(INKFLocale.LEVEL_DEBUG,"SOURCE argument = " + lThisRequest.getArgumentName(i));
      if (! "accessorname".equals(lThisRequest.getArgumentName(i))) {
        subrequest.addArgument(lThisRequest.getArgumentName(i), lThisRequest.getArgumentValue(i));
      }
    }
    
    if ("GET".equals(aHTTPMethod)) {
      subrequest.setVerb(INKFRequestReadOnly.VERB_SOURCE);
    }
    else if ("POST".equals(aHTTPMethod)) {
      subrequest.setVerb(INKFRequestReadOnly.VERB_NEW);
    }
    else if ("PUT".equals(aHTTPMethod)) {
      subrequest.setVerb(INKFRequestReadOnly.VERB_SINK);
    }
    else if ("DELETE".equals(aHTTPMethod)) {
      subrequest.setVerb(INKFRequestReadOnly.VERB_DELETE);
    }
    else if ("HEAD".equals(aHTTPMethod)) {
      subrequest.setVerb(INKFRequestReadOnly.VERB_EXISTS);
    }
    else {
      subrequest.setVerb(INKFRequestReadOnly.VERB_SOURCE);
    }


    // One response
    aContext.createResponseFrom(aContext.issueRequestForResponse(subrequest));
  }

That was not too hard. The RESTMethodToVerbAccessor works as a dispatcher, doing the method-to-verb translation and launching the real request with the correct verb.

There's one thing I'm not too happy about. For the RESTOverlay the inner grammar has to be unique. Hence the active:restMethodToVerb_pp and active:restMethodToVerb_gd identifiers. That is not elegant. For me the inner grammar + method would have to be unique.

Let us start the discussion. I've created a topic on the NetKernelROC forums where we can have a free-for-all (discussion that is). 

logging

It has been brought to my attention that I can write posts here on practically anything  that is closely or remotely linked to NetKernel and even IT in general. Today I want to discuss something with you that has been a thorn in my side for years. Logging.

Some 19 years ago, when I - fresh out of school - started in IT, I also started getting daily listings (on paper) with loggings. I would get an overview of the program-identifiers I requested, another list would be almost identical but contain file-identifiers and so on.

My listings were peanuts compared to the ones the old hands got. In fact, you could easily determine the status/position of someone by looking at the amount of listings with loggings he/she got. The higher the position, the higher the pile.

Many a lament of many a tree went unheard as it was cut down to support this habit. And of course, most of those listings got no more than a single glance before being thrown away.

IT grew up, companies became environmentally aware, I started doing database and system administration, paper listings had to go, mails and chats and log consoles ... replaced them. Until this very day I come in in the morning, take one look at the loggings and run to the toilet or the coffee machine (or both, order may depend upon the quality of the coffee).

Please, please, log as sparsely as you can :
Why ? Overhead is overhead. Even if it is small.
Why ? Storage costs money. Even if it is not a lot. Do you really want to have more loggings about your data (often repeating the data) than you have data ?
Why ? Well, at the end of the day somebody has to look at all that data. And I've spend my share of nights sitting next to the console jockey (I was there for a rollout of something usually and ... I admit, they had better coffee than we had) wading through all kinds of messages popping up. I also sat next to the audit person going through one nights worth of database changes/logins/logouts to find the culprit of a certain rather clever (not so clever that I didn't notice) change on a top secret database.

NetKernel has a nice logging system. By all means check it out and use it. Logging in NetKernel is asynchronously and used in moderation will have a negligible impact on your application. Do spare a thought though for the poor human that will have to check them !

P.S. If you did come here to learn about swamp logging, check here.

space listing - the future - part 1

My schedule has been a little hectic the last couple of weeks, but this afternoon I finally found time to revisit the Space Listing. So, without further ado, here's the rootspace that contains my version of it :

<rootspace

  name="System Accessors"

  public="true"

  uri="urn:org:elbeesee:ext:system:accessors">

    

  <accessor>

    <id>system.SpaceStaticResourceList.accessor</id>

    <class>org.elbeesee.ext.system.SpaceStaticResourceListAccessor</class>

    <grammar>

      <active>

        <identifier>active:ssrls</identifier>

        <argument name="space" desc="space identifier (urn)"/>

        <argument name="version" min="0" max="1" desc="space version"/>

      </active>

    </grammar>

  </accessor>

    

  <fileset>

    <regex>res:/resources/stylesheets/.*</regex>

  </fileset>

  <import>

    <uri>urn:org:netkernel:xml:core</uri>

    <private/>

  </import>

    

  <import>

    <uri>urn:org:netkernel:ext:layer1</uri>

    <private/>

  </import>

</rootspace>

Nothing special there, same definition as the original active:sls. It's in the code that things are different, so lets have a look at that :

package org.elbeesee.ext.system;

// Author: Tom Geudens

// Date  : 2012/04/08

// The usual suspects for an accessor.

import org.netkernel.layer0.nkf.*;

import org.netkernel.layer0.meta.impl.SourcedArgumentMetaImpl;

import org.netkernel.module.standard.endpoint.StandardAccessorImpl;

// Processing.

import java.io.File;

import java.net.URI;

import java.util.Iterator;

import org.netkernel.container.IKernel;

import org.netkernel.module.standard.StandardSpace;

import org.netkernel.layer0.boot.BootUtils;

import org.netkernel.layer0.representation.IHDSNode;

import org.netkernel.layer0.representation.impl.HDSBuilder;

import org.netkernel.layer0.urii.SimpleIdentifierImpl;

import org.netkernel.urii.ISpaceWithIdentity;

import org.netkernel.urii.impl.Version;

public class SpaceStaticResourceListAccessor extends StandardAccessorImpl {

  public SpaceStaticResourceListAccessor() {

    this.declareThreadSafe();

    this.declareArgument(new SourcedArgumentMetaImpl("space",null,null,new Class[] {String.class}));

    this.declareArgument(new SourcedArgumentMetaImpl("version",null,null,new Class[] {String.class}));

  }

  private IHDSNode listDirectory(INKFRequestContext aContext, File aDirectoryFile) {

    return listDirectory(aContext, aDirectoryFile, "res:/");

  }

  private IHDSNode listDirectory(INKFRequestContext aContext, File aDirectoryFile, String aResourcePath) {

    HDSBuilder lDirectoryBuilder = new HDSBuilder();

    File[] lDirectoryFiles       = aDirectoryFile.listFiles();

    lDirectoryBuilder.pushNode("resources");

    for (int i=0; i < lDirectoryFiles.length; i++) {

      if (lDirectoryFiles[i].isDirectory()) {

        String lResourcePath = aResourcePath + lDirectoryFiles[i].getName() + "/";

        lDirectoryBuilder.importChildren(listDirectory(aContext,lDirectoryFiles[i],lResourcePath).getFirstNode("/resources"));

      }

      else {

        lDirectoryBuilder.addNode("resource", aResourcePath + lDirectoryFiles[i].getName());

      }

    }

    lDirectoryBuilder.popNode();

    return lDirectoryBuilder.getRoot();

  }

  public void onSource(INKFRequestContext aContext) throws Exception {

    // One mandatory argument

    String aSpaceIdentifier = aContext.getThisRequest().getArgumentValue("space");

    

    // One optional argument

    Version aVersion = null;

    if (aContext.getThisRequest().argumentExists("version")) {

      aVersion = new Version(aContext.getThisRequest().getArgumentValue("version"));

    }

    

    // Processing

    IKernel lKernel = null;

    ISpaceWithIdentity lSpace = null;

    StandardSpace lSS = null;

    String lSource;

    

    lKernel = aContext.getKernelContext().getKernel();

    lSpace  = lKernel.getSpace(new SimpleIdentifierImpl(aSpaceIdentifier), aVersion, aVersion);

    

    if (lSpace == null) {

      throw new NKFException("Space is not found");

    }

    

    if (!(lSpace instanceof StandardSpace)) {

      throw new NKFException ("Space is not a standard module space");

    }

    

    lSS = (StandardSpace)lSpace;

    lSource = lSS.getOwningModule().getSource().toString();

    lSource = BootUtils.fixURIString(lSource);

    

    IHDSNode lModuleXML = null;

    IHDSNode lRootSpace = null;

    if (lSource.startsWith("file:")) {

      lModuleXML = aContext.source(lSource + "module.xml",IHDSNode.class);

    }

    

    INKFRequest subrequest = aContext.createRequest("active:xslt");

    subrequest.addArgumentByValue("operand", lModuleXML);

    subrequest.addArgument("operator", "res:/resources/stylesheets/module.xsl");

    subrequest.addArgumentByValue("spaceid", aSpaceIdentifier);

    subrequest.setRepresentationClass(IHDSNode.class);

    

    lRootSpace = (IHDSNode)aContext.issueRequest(subrequest);

    File lSourceFile = new File(URI.create(lSource));

    IHDSNode lSourceContent = listDirectory(aContext,lSourceFile);

    

    HDSBuilder lValidResources = new HDSBuilder();

    lValidResources.pushNode("resources");

    Iterator<IHDSNode> lIterator = lRootSpace.getNodes("/rootspace/fileset").iterator();

    while(lIterator.hasNext()) {

      IHDSNode lFileset = lIterator.next();

      String lGlob  = null;

      String lRegex = null;

      lGlob  = (String)lFileset.getFirstValue("glob");

      lRegex = (String)lFileset.getFirstValue("regex");

      if (lGlob != null) {

        System.out.println("fileset glob = " + lGlob);

      }

      if (lRegex != null) {

        Iterator<IHDSNode> lIterResource = lSourceContent.getNodes("/resources/resource").iterator();

        while (lIterResource.hasNext()) {

          IHDSNode lResource = lIterResource.next();

          String lResourceString = (String)lResource.getValue();

          if (lResourceString.matches(lRegex)) {

            lValidResources.addNode("resource", lResourceString);

          }

        }

      }

    }

    lValidResources.popNode();

    // One response

    INKFResponse response = null;

    // response = aContext.createResponseFrom(lRootSpace.getRoot());

    // response = aContext.createResponseFrom(lSourceContent.getRoot());

    response = aContext.createResponseFrom(lValidResources.getRoot());

  }

}

What happens here ?
1) Determine the module of the rootspace that is passed as an argument.
2) Extract the rootspace from the module.xml of that module.
3) Determine all possible file resources of the module.
4) Loop through the filesets defined in the rootspace and use those to filter the file resources so only the valid ones remain.

This is the stylesheet that is used to extract the rootspace from module.xml :

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

                xmlns:nk="http://netkernel.org" 

                version="1.0">

  <xsl:output method="xml" 

              indent="yes" 

              encoding="UTF-8" 

              omit-xml-declaration="yes"/>

  <xsl:param name="spaceid" nk:class="java.lang.String" />

  <xsl:template match="/module">

    <xsl:copy-of select="rootspace[@uri=$spaceid]"/>

  </xsl:template>

</xsl:stylesheet>

And ... done.

Or so it seems, you might have noticed I labeled this entry "part 1". There's some work to be done yet :
* Next to regex, a fileset can also be glob or grammar defined. All three should be handled (at the moment only regex is).
* A fileset definition can contain a rewrite. I want to get the real valid resources, so we need to provide a rewrite too.
* A module can be expanded or jarred. At the moment the accessor only works for expanded modules. I want jars too.

As you can see there's room for a "part 2" (and beyond ?). Watch this space ! There might be other requirements you are interested in. Let me know and I'll add them.

Since this was a lot of code real fast, my next blog entry will once again be a philosophical one. You've been warned ! 

troubleshooting

Halfway through the week I realised that I might have overreached myself just a little by promising to deliver the future and then most of the fun I had this week was still to come.

So, I'll postphone the future to next week and give you a short troubleshooting tip this week. For sometimes things go wrong. Networks can flap, disks can fail, processors can overheat. And then there's the OS, the JVM, ... Sometimes it is just amazing that anything works at all.

There is of course the Status tab in the HTTPBackend Fulcrum that has live graphs to study, but there is also the Requests and Threads page in the Developer tab.

Now this is a static page and I needed the information on just one specific thread at a more frequent interval. So I wrote a new tool :

<mapper>
  <config>
    <endpoint>
      <grammar>res:/tools/kerneldetail/
        <group name="threadname">
          <regex type="anything"/>
        </group>
      </grammar>
      <request>
        <identifier>active:xslt</identifier>
        <argument name="operand">netkernel:/k</argument>
        <argument name="operator">res:/resources/stylesheets/kerneldetail.xsl</argument>
        <argument name="threadname" method="as-string">arg:threadname</argument>
      </request>
    </endpoint>
  </config>
  <space>
    <fileset>
      <regex>res:/resources/stylesheets/.*</regex>
    </fileset>
    <import>
      <uri>urn:org:netkernel:xml:core</uri>
      <private/>
    </import>
    <import>
      <uri>urn:org:netkernel:ext:system</uri>
      <private/>
    </import>
  </space>
</mapper>

Did I say write ? No code (after no sql it is now time for the new paradigm, no code) of course. The netkernel:/k resource provides the raw data and I filter that with xslt :
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:nk="http://netkernel.org" 
                version="1.0">
  <xsl:output method="xml" 
              indent="yes" 
              encoding="UTF-8" 
              omit-xml-declaration="yes"/>
  <xsl:param name="threadname" nk:class="java.lang.String" />


  <xsl:template match="/k">
    <xsl:copy-of select="threads/thread[name=$threadname]"/>
  </xsl:template>
</xsl:stylesheet>

And that's it. Put in a small script ... for example like this :
#!/bin/sh
while true
do
 curl http://localhost:1060/tools/kerneldetail/ConcurrentCacheCullThread >> /var/tmp/culler.out
 sleep 10
done
And collect all the data you want with minimum system impact.

So, next week I will deliver the future as promised.

space listing

I remember being disappointed when I learned that an asteroid belt is nothing like what I knew from videogames (also an excellent song by Lana Del Rey), where you have to maneuver your ship between rocks with hardly any space between them. The truth is that you can pass through an asteroid belt without even seeing or coming close to any asteroid at all!

Space is like that. Contains a near infinite number of things but if you don't know where to look it might as well be empty.

Spaces in ROC (because of the way I set up your mind you might read that as "rocks in space") can also be like that. Sometimes you just want to know what is there before you go in and request it.

Note that this is not a logical train of thought. Because - and I quote Peter Rodgers - how do you enumerate a potentially infinite set?

It is however a practical train of thought, one that can only be followed through for spaces that are enumerable (whatever that means for the given space type). Such as a fileset. And of course, there's a tool in NetKernel that allows you to do that. Try the following in the Scripting Playpen :

<sequence>

  <request assignment="response">

    <identifier>active:sls</identifier>

    <argument name="space">urn:org:netkernel:ext:system</argument>

  </request>

</sequence>


Here's a snippet of the result :

<dir id="res:/etc/" name="etc">

  <res id="res:/etc/messages.properties" name="messages.properties"/>

  <dir id="res:/etc/system/" name="system">

    <res id="res:/etc/system/Books.xml" name="Books.xml"/>

    <res id="res:/etc/system/Docs.xml" name="Docs.xml"/>

  </dir>

</dir>


Now we are getting somewhere! This opens up possibilities for a resource oriented ftp server or a resource oriented content management system or a <your idea here>.

When you read the documentation for sls you'll notice that there are plans for some future enhancements. Come and see the future in next week's entry!

What did you expect ?

One of these days Peter Rodgers is bound to mention in the NetKernel Newsletter that I repeat/rehash a lot of what he and Tony have been saying all along. Of course I do. What did you expect ?

I want to focus on that question today. For in these days of multi-core systems, cheap memory (buy lots) even cheaper storage (buy lots lots) and unlimited bandwidth that question is often not even thought of until an application is going through the final performance tests.

And at that point your company's performance guy (been there, done that) might say - bad - things like :
- Those response times are - imo - not acceptable for production, what did you expect ?
- That is a very high throughput you've got there, what did you expect ?
- The cpus on the system are going red-hot, what did you expect ?

With a deadline looming like the sword of Damocles above your head, these are not things you want to hear.

The question "What do you expect ?" should be asked a lot earlier, it should be asked in the design phase, before any coding is done. The cost of overhauling a design is a lot lower than the cost of overhauling a completely finished application.

Now, next to the fact that NetKernel has an extremely light footprint on your systems as well as superior caching, it also has the tools to help you stay within the limits of what you expect.

Take nkperf. You can install this tool from the repository and it tells you how your system behaves. Not relevant you say ? Read this newsletter entry again. Or if you'd rather not have a rehashed ;-) story, look at what I found this morning on my own laptop (Windows, dual core intel) :

Oops, there is something very fishy with my caching performance. Especially when you look at this :

The second graph is from a virtual machine (CentOS, single core), running on the same laptop !

So why is the blue line flat on the first graph ? I don't know, but I better find out before I make any conclusions about applications running on that system. And that is the point of my story. You must know what you expect, what the constraints on your application are before you write it. And NetKernel can help you with that.


Footnote :
On seeing the above issue, the 1060 Research crew jumped in and pointed out the obvious cause of the flatliner. I've got a pretty speedy laptop. And to optimize caching, NetKernel has a cost threshold parameter described as : cost of an item must be at least this high before eligible for caching. Can you see where this is going ? The native machine (but not the vm) responded so fast that nothing was costly enough to be cached. Hence the flatliner.

Easily ammended, I set the cost threshold to 0. And look :