2013/12/31

interesting times

If you're looking for the episode on logging, soon my friend, soon. 
If you've noticed my last post of 2012 was also titled interesting times, you are correct.

A year ago I promised Tony Butterfield that I would force a breakthrough for Resource Oriented Computing / NetKernel in 2013. With about eight hours left in 2013 here in Brussels, Belgium, I must admit defeat as far as that claim goes.

People that know me personally know that I do not take defeat - in any area - lightly or graciously. 

It has been a good year but not the exceptional year that I envisioned when I made that claim.

So, here are some of my ROC related targets for 2014 :
  • At least one ROC backed game playable in one of the mainstream social networks.
  • Positioning ROC as the best solution for Linked (Open) Data projects.
  • Positioning ROC as the green IT solution. With the point of no return for climate change set in 2015, computing must start contributing too.
  • A new book.
  • <any target you, my constant readers, care to set ?>
  • ...
Best wishes and see you in 2014 !
 

2013/12/11

back to the beginning ... unit test

In IT a lot of fuss is made these days around testing and logging. The former is used to avoid introducing bugs into software and the latter is used to be able to find the inevitable bugs that are introduced into software.

And no, the above is not said in a light or satirical tone. I wouldn't dare, the industry has gotten quite anal on these subjects, as - to give one example - Rick Hickey found out when he compared Test Driven Development (TDD) to learning to drive by bumping into the guardrails. Testing and logging are to be taken very serious and they have become big moneymakers in their own right.

This does say a lot about the IT industry as a whole, but I'm not going there either ...

Today I'm going to discuss testing within the Resource Oriented Computing worldview, logging will be in the spotlight next time.

Why test ?
  • unit test : to check if a piece of code delivers on its promise 
  • integration / system test : to check if a piece of code works in combination/interaction with other pieces of code in a complex system
  • regression test : to check if a piece of code still works after a change has been introduced elsewhere in a complex system 
You probably have other definitions in your textbook, mine go back to my initial IT training in the early nineties. Yes, I am that old.

When you're working with ROC/NetKernel, you'll soon notice that there's less fuss about testing. Reasons :
  • less code
  • smaller components
  • less complexity
Another - maybe less obvious - reason is that constraints are typically applied at the end of the Resource Driven Development (RDD) cycle - more on Construct Compose Constrain in a later post - not at the beginning. RDD and TDD approach the same problems from opposite sides.

Time to discuss the testing framework in NetKernel. Yes there is one and yes, it works with resources. Hey, what did you expect ? Last time I created a service for the Amazon Web Service tools. You can find the source for that module, urn.com.amazonaws.sdk on Github.

You'll find three spaces in the module.xml. The public urn:com:amazonaws:sdk space that contains the service, the private urn:com:amazonaws:sdk:import space that contains the internally used resources and tools and the public urn:com:amazonaws:sdk:unittest space. Can you guess what that last one is for ?

There is an ongoing debate whether or not the unit tests should be included in a module or live in a separate module. As with the naming of spaces (I've gotten some comments on that) ... whatever suits you best. If you are adamant about not deploying tests to production, you might want to split them off. If you want to have tests for your private spaces - as I do - you must include them.

The testing framework works - like the fulcrums - with a dynamic import. It searches for public spaces with a res:/etc/system/Tests.xml resource and pulls those spaces inside.

It is a bit confusing given the tag-names, but the resource describes groups (!) of tests. Some people prefer one group for the whole module, I usually have one group per space that I want to test :
            <tests>
                <test>
                    <id>test:amazonaws:sdk:import</id>
                    <name>amazonaws sdk import unittest</name>
                    <desc>amazonaws sdk import unittest</desc>
                    <uri>res:/resources/unittest/sdk-import.xml</uri>
                </test>
                <test>
                    <id>test:amazonaws:sdk</id>
                    <name>amazonaws sdk unittest</name>
                    <desc>amazonaws sdk unittest</desc>
                    <uri>res:/resources/unittest/sdk.xml</uri>
                </test>
            </tests>

 

Each group contains the uri of the resource with the actual list of tests for that group, in this case res:/resources/unittest/sdk-import.xml and res:/resources/unittest/sdk.xml. Lets look at the former :
            <testlist>
                <test name="SOURCE regions.xml resource">
                    <request>
                        <identifier>res:/resources/xml/regions.xml</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <notNull/>
                    </assert>
                </test>
            </testlist>

 

And the latter :
            <testlist>
                <test name="SOURCE s3 eu-west-1 resource">
                    <request>
                        <identifier>res:/amazonaws/baseurl/s3-eu-west-1</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <stringEquals>s3-eu-west-1.amazonaws.com</stringEquals>
                    </assert>
                </test>
            </testlist>


These test-lists can contain as many tests as you deem necessary. In this case I have one in each. I check if regions.xml is there and I check one case of the service.

If you've been properly indoctrinated you'll probably wince in agony at the above. Only two tests ? What about some more cases ? Bear with me, if you want to add more, you can.

Let us have a look at the XUnit Tests tool first.

I filtered the results so you only see the two relevant test-lists. If you want to run a lot of tests at once, select the test-lists you want to run :

and select execute :

Or you can select one test-list with view :

and execute that :

Note that this is not only a GUI, it is quite easy to add this to an automation and get the test-results as XML documents. Everything is a resource !

I'm not going to run through all the XUnit Testing documentation, but lets add a couple more tests so you all feel a bit more at ease. First in res:/resources/unittest/sdk-import.xml :
            <testlist>
                <test name="SOURCE regions.xml resource">
                    <request>
                        <identifier>res:/resources/xml/regions.xml</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <notNull/>
                    </assert>
                </test>
 
                <test name="SOURCE regions.dat resource">
                    <request>
                        <identifier>res:/resources/xml/regions.dat</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <exception/>
                    </assert>
                </test>

            </testlist>
 
If you then run the tests for that list this is what you get :

and next in res:/resources/unittest/sdk.xml :
            <testlist>
                <test name="SOURCE s3 eu-west-1 resource">
                    <request>
                        <identifier>res:/amazonaws/baseurl/s3-eu-west-1</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <stringEquals>s3-eu-west-1.amazonaws.com</stringEquals>
                    </assert>
                </test>
 
                <test name="SOURCE s4 eu-west-1 resource">
                    <request>
                        <identifier>res:/amazonaws/baseurl/s4-eu-west-1</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <exception/>
                    </assert>
                </test>

             </testlist>

If you then run the tests for that list this is what you get :

Oops ... that wasn't quite what I intended. What went wrong here ? Select exec to see the actual result of the request :
The grammar of the res:/amazonaws/baseurl/s4-eu-west-1 request is valid, so the regions.xml resource is passed through the xslt transformation and comes up with unknown as the result. A valid result. Not an exception. So lets change our test-list again :
            <testlist>
                <test name="SOURCE s3 eu-west-1 resource">
                    <request>
                        <identifier>res:/amazonaws/baseurl/s3-eu-west-1</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <stringEquals>s3-eu-west-1.amazonaws.com</stringEquals>
                    </assert>
                </test>
 
                <test name="SOURCE s4 eu-west-1 resource">
                    <request>
                        <identifier>res:/amazonaws/baseurl/s4-eu-west-1</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <
stringEquals>unknown</stringEquals>
                    </assert>
                </test>

                <test name="SOURCE blabla resource">
                    <request>
                        <identifier>res:/amazonaws/baseurl/blabla</identifier>
                        <verb>SOURCE</verb>   
                    </request>
                    <assert>
                        <exception/>
                    </assert>
                </test>

            </testlist>

And the result :

Much better. And here's my opinion : the extra test I added to res:/resources/unittest/sdk-import.xml is useless as it doesn't test functionality, the two extra tests I added to res:/resources/unittest/sdk.xml are useful as they do test functionality. It may not always be so clear and if you feel you've got to err on the side of caution, by all means do. Just remember that your purpose is solving problems, not writing tests.

Right, that's a lot of information again. Don't forget to look through the XUnit Test documentation, you'll find that you can adapt and extend it to your own preferences.

As I promised, next time I'll discuss logging and as a bonus I'll also explain what that Limiter endpoint (did you notice it in the unit test space ?) does. Keep on ROCing in the mean time !

2013/11/29

back to the beginning ... mapper pattern

It was to be expected, the moment I try to take things from the top, there is a major update to the NetKernel administrative interface (aka Backend Fulcrum). If you haven't done the update yet, you should, it is astonishing. I personally have a blind spot in the user experience area (being a command line jockey) but even I can see this is a huge improvement.
No, I'm not going to update my prior posts with new screen shots. Things have been cleaned up and integration between the tools is now much tighter. However, there is no change to what lies underneath.

An overhaul of that magnitude would in most products warrant a major release change, however, the crew @ 1060 Research is too honest and rated the two-week effort with only a minor release. 

In my previous three posts I looked at the core of Resource Oriented Computing. In this and couple more posts I am going to discuss a couple of very common patterns. Today the spotlight is on the mapper.

== Intermezzo (you can safely skip this) BEGIN ==

Before I embark on that quest - creating a real service as I do so - I want to spend a moment on what is known as the magic/silver bullet fallacy. Also known as the deus ex machina fallacy :
  • There is no such thing as a free lunch (no, not even in the Google/Apple/... offices). You can safely ignore any paradigm that claims it will solve problems effortlessly. That goes against the second law of thermodynamics. ROC claims no such thing. What it does claim is that it allows you to solve problems a lot more efficiently !
  • I can explain it to you but I can't understand it for you. I got this bon mot from the Skype status of a very smart developer. Ironically he didn't quite understand ROC, but that is not the point. The point is that effort is involved.
Why am I saying this ? Because the service I'm going to create in this blog involves the use of an XSLT transformation and stylesheet. Maybe you master this technology, maybe you don't. You don't have to master it to follow the example, but my point is that you need some tools in your IT toolbelt. NetKernel comes with a lot of tools in the box (and you can add your own too). It will take effort to master them.
== Intermezzo (time to pay attention again) END ==

Amazon Web Services has grown into a large platform with a plethora of services (ec2, s3, ...). You can integrate these into your own solutions using the SDK or an HTTP service. In the latter case, each service has its own URL, which also depends on the region that you want to use the service in.

I'm ignoring the SDK for now. Integrating 3rd party libraries is a topic for another post.

In order to keep track of all these URLs, Amazon has published a regions.xml resource. As you can see, it contains all the URLs for all the services in all the regions. Woot !

But what if I just want to know what the Hostname is for the S3 service in the eu-west-1 region ? Well, that's the service I am going to build. It will define this resource : res:/amazonaws/baseurl/{service}-{region}.

For a first iteration we are going to download the regions.xml file and include it into our module itself as a file resource, res:/resources/xml/regions.xml.
    <rootspace
        name="amazonaws sdk import"
        public="false"
        uri="urn:com:amazonaws:sdk:import">
       
        <fileset>
            <regex>res:/resources/xml/.*</regex>
        </fileset>
    </rootspace>


The regions.xml file is an XML document. Processing an XML document can be done in many ways, I like using XSLT which takes the data and applies a transformation stylesheet to it. So we are also going to require a stylesheet, regions.xsl, which is also going to be a file resource, res:/resources/xsl/regions.xsl. We could define another fileset for that, but with a small change to the above we can cater for both :
    <rootspace
        name="amazonaws sdk import"
        public="false"
        uri="urn:com:amazonaws:sdk:import">
       
        <fileset>
            <regex>res:/resources/(xml|xsl)/.*</regex>
        </fileset>
    </rootspace>


The stylesheet itself looks like this :
<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:nk="http://1060.org"
    exclude-result-prefixes="nk">
   
    <xsl:output
        method="text"
        encoding="UTF-8"
        media-type="text/plain"/>

    <xsl:param name="service" nk:class="java.lang.String"/>
    <xsl:param name="region" nk:class="java.lang.String"/>
   
    <xsl:template match="/XML">
        <xsl:if test="Regions/Region[Name=$region]/Endpoint[ServiceName=$service]/Hostname">
            <xsl:value-of select="Regions/Region[Name=$region]/Endpoint[ServiceName=$service]/Hostname"/>
        </xsl:if>
        <xsl:if test="not(Regions/Region[Name=$region]/Endpoint[ServiceName=$service]/Hostname)">
            <xsl:text>unknown</xsl:text>
        </xsl:if>
    </xsl:template>
</xsl:stylesheet>


There, that's that. NetKernel comes with a lot of tools for processing and transforming data and since its roots lie in the XML era, it is no surprise that an XSLT transformation engine is readily available. It is part of the xml:core toolset. Including xml:core changes the above space to this :
    <rootspace
        name="amazonaws sdk import"
        public="false"
        uri="urn:com:amazonaws:sdk:import">
       
        <fileset>
            <regex>res:/resources/(xml|xsl)/.*</regex>
        </fileset>
       
        <import>
            <!-- active:xsltc -->
            <uri>urn:org:netkernel:xml:core</uri>
        </import>
    </rootspace>


Right, as you probably deduced, I've put our bill-of-material in a private space, as I don't want any of it publicly available. Before we get to the public part, lets see for a moment what we can do in this space :
The full declarative request is :
<request>
    <identifier>active:xsltc</identifier>
    <argument name="operand">res:/resources/xml/regions.xml</argument>
    <argument name="operator">res:/resources/xsl/regions.xsl</argument>
    <argument name="region">data:text/plain,eu-west-1</argument>
    <argument name="service" >data:text/plain,s3</argument>
    <representation>java.lang.String</representation>
</request>


And no, I haven't done anything strange or new with that (although it is a new - and very welcome - feature of the Resource Trace Tool), it is the equivalent of :
active:xsltc+operand@res:/resources/xml/regions.xml+operator@res:/resources/xsl/regions.xsl+region@data:text/plain,eu-west-1+service@data:text/plain,s3

Which is certainly a level up from a simple resource request like res:/resources/xml/regions.xml (try that first if you're uncomfortable with the above) but you can see how it is actually merely pulling together four resource requests as input for the active:xsltc resource request. This process is called composing.

If you've looked at the documentation for active:xsltc, you're might wonder why I have the extra "region" and "service" arguments. Well, they are varargs (variable arguments) which in this case are "optional parameters provided to the stylesheet runtime". You'll find them used in the stylesheet.
 
What is left is to map our proposed resource res:/amazonaws/baseurl/{service}-{region} to the above request in a public space :
    <rootspace
        name="amazonaws sdk"
        public="true"
        uri="urn:com:amazonaws:sdk">
       
        <mapper>
            <config>
                <endpoint>
                    <grammar>
                        <simple>res:/amazonaws/baseurl/{service}-{region}</simple>
                    </grammar>
                    <request>
                        <identifier>active:xsltc</identifier>
                        <argument name="operand">res:/resources/xml/regions.xml</argument>
                        <argument name="operator">res:/resources/xsl/regions.xsl</argument>
                        <argument method="as-string" name="region">[[arg:region]]</argument>
                        <argument method="as-string" name="service">[[arg:service]]</argument>
                        <representation>java.lang.String</representation>
                    </request>
                </endpoint>
            </config>
            <space>
                <import>
                    <uri>urn:com:amazonaws:sdk:import</uri>
                    <private/>
                </import>
            </space>
        </mapper>
    </rootspace>


While this may not all be immediately clear, it should be clear what happens. An incoming resource request of the form res:/amazonaws/baseurl/{service}-{region} gets mapped to an active:xsltc resource request which is issued into the wrapped space. The response representation goes the other way.

The method="as-string" attribute turns values into resources. Don't worry about that too much at this point. I could have specified those arguments as follows :
   <argument name="region">data:text/plain,[[arg:region]]</argument>
   <argument name="service">data:text/plain,[[arg:service]]</argument>

but then I'd have had to import the Layer1 space as I explained in my post about spaces.

Check out the result :

If you remember from last time (extra credits to you !) how to make this available in your browser as http://localhost:8080/amazonaws/baseurl/s3-eu-west-1 ... you've just created a real and useful service. Congratulations !


Having regions.xml as a file in your module is fine, but it would be better if we could use the regions.xml that Amazon has published. You can (this will require internet access to work). First replace 
  <argument name="operand">res:/resources/xml/regions.xml</argument>
with
  <argument name="operand">https://raw.github.com/aws/aws-sdk-java/master/src/main/resources/etc/regions.xml</argument>

Then redeploy the module and check out the result :

The problem is that your space doesn't know yet how to resolve a http:// resource. Not a problem though, NetKernel has a toolkit available that does know :
    <rootspace
        name="amazonaws sdk import"
        public="false"
        uri="urn:com:amazonaws:sdk:import">
       
        <fileset>
            <regex>res:/resources/(xml|xsl)/.*</regex>
        </fileset>
       
        <import>
            <!-- active:xsltc -->
            <uri>urn:org:netkernel:xml:core</uri>
        </import>
       
        <import>
            <!-- http:// scheme  -->
            <uri>urn:org:netkernel:client:http</uri>
        </import>

    </rootspace>


Redeploy, try again and you'll get the same result as before.


Summary
  • We've looked at an interesting pattern today, the mapper pattern. 
  • We've composed resources together into new resources, we wrote no custom code.
  • We've made use of tools provided in NetKernel.
  • We have build a real service.
It is a lot of information and you may feel that I'm taking a huge leap forward from where I left off last time, but do take it one step at a time and you'll see that is not the case. Enjoy !

2013/11/16

back to the beginning ... transports

Today I'm making good on my promise. We're going to make the res:/introduction/helloworld-xxxxx resources available in your browser. If the host you are running NetKernel on is open to the Internet, we might even make them available to the whole world !

I've had some questions about it so I want to repeat one point I made. The last two posts did cover the core of Resource Oriented Computing ! There really is no more. You do have all the building blocks and not unlike those in Lego they are simple. There is however no - practical - limit to what you can build with them.


Before we start making changes to our introduction module, I first have to explain about transports. NetKernel provides a Resource Oriented ... abstraction or box or - my personal view as a former System Administrator - operating system and until now we were only able to request the resources inside that with the Request Resolution Trace Tool. I bet that is not how you want to run your software. Transports solve this problem. A transport listens for incoming requests and brings them inside. The response (resource representation) goes the other way.

That may sound complex but again you are most likely already familiar with this concept. If you're a database administrator, you might know that an Oracle server listens on port 1521 for incoming database requests. The Apache server listens on port 80 for incoming http:// requests. The ssh daemon listens on port 22 for incoming ssh requests. Und so weiter.

As you can see thinking of NetKernel as an operating system was not so silly, for indeed, the transports it comes with work in exactly the same way. NetKernel has several transports available, out-of-the-box it has two running. On port 8080 and port 1060 it has Jetty (alternative for Apache) server based transports listening for incoming http:// requests.

The transport on port 8080 is also known as the FEF - Front End Fulcrum, the one on port 1060 as the BEF - Back End Fulcrum (or Administrative Fulcrum). For me personally this last acronym is confusing as BEF also stands for Belgian Franc (obsolete since 2002), I guess my age is starting to show.

You may already be able to deduce where this is going, these transports translate incoming http:// requests into res:/ requests.

For example :
  1. You request http://localhost:1060/tools/requesttrace in your browser.
  2. The transport listening on port 1060 translates this into a res:/tools/requesttrace request.
  3. The res:/tools/requesttrace resource can be found in the urn:org:netkernel:ext:introspect space (no worries about how I found that out).
  4. Therefore it must follow that the space that contains the transport - that would be urn:org:netkernel:fulcrum:backend - has access to (imports) the urn:org:netkernel:ext:introspect space.
We are going to proof the above by making our res:/introduction/helloworld-xxxxx resources available to the FEF.

We could just as well do it on the BEF, go ahead and try that as an exercise if you like !

The FEF lives in [installationdirectory]/modules/urn.org.netkernel.fulcrum.frontend-1.5.12. Change the module.xml file in there to import the urn:org:elbeesee:experiment:introduction space. Where ? Well, right underneath the comment :

  <!--Import application modules here-->
  <import>
    <uri>urn:org:elbeesee:experiment:introduction</uri>
  </import>

Save and try http://localhost:8080/introduction/helloworld-file in your browser.
Something went wrong. If you've been very attentive you may have noticed that the module.xml file of the FEF does not contain a <dynamic/> tag, changes to the module.xml file are not picked up. Stop and start NetKernel and try again.
Success. We have proved the concept.

However, if you're anything like me, it doesn't feel like success. Sure, it works, but that's quite a process to go through. Changing a file and stopping and starting NetKernel to deploy a resource to the FEF ? Anno Domini 2013 ? Really ? In fact, it was the process to follow in NK3. Things have changed.

You may undo your changes to FEF module.xml, stop and start NetKernel again (the introduction urls shouldn't work any longer) and then I want to draw your attention to this snippet :

  <!--Dynamic Imports-->
  <endpoint>

    <prototype>SimpleImportDiscovery</prototype>
    <grammar>active:SimpleImportDiscovery</grammar>
    <type>HTTPFulcrum</type>
  </endpoint>


What does that mean ? Well, it means that the FEF will automatically import spaces that contain a certain resource. So instead of changing the FEF, we're going to change our own module and add the following to the
urn:org:elbeesee:experiment:introduction space :

  <literal type="xml" uri="res:/etc/system/SimpleDynamicImportHook.xml">
     <connection>
       <type>HTTPFulcrum</type>
     </connection>
  </literal>

The uri and content of the resource are what the SimpleImportDiscovery endpoint expects. You can also provide the resource in any of the other forms that we have seen (a file resource is very common).

And you'll notice the introduction urls work once more.

Not only is this very convenient (and we take that for granted in this day and age), it is also a very powerful pattern that you can use for your own solutions !

This brings me to the end of this post. Before I go I'd like to make an important observation. It is very easy to forget - I know I did for quite a while - that other than the default (http://) transports can be used to trigger resource requests. There's a cron-transport. An ssh-transport. A mail-transport. And if all of these available transports don't fill your needs, it has been shown that any of the Apache Camel Components can be turned into a custom transport for NetKernel.

2013/11/08

back to the beginning ... spaces

I got some very nice feedback on my previous post (thank you, you know who you are), so I'm starting this time with the assumption that there is demand for basic information. Today we are taking a closer look at spaces.

You are almost certainly familiar with the concept. If you are into mathematics, you might consider a space as a set. If you are into programming, you might consider a space as a scope. The very fact that you are reading this post, means that you have issued a http:// request into a space that knows how to resolve such requests ... a space better known as the World Wide Web. You may or may not understand in detail how it (the World Wide Web) works but you certainly issued that request with a pretty good idea of what you were going to get.
Let us look at one such request :
  1. http://netkernelbook.org/author.html is requested in the World Wide Web space, you can do that with a - almost - magic tool called browser.
  2. resolution takes place, in the case of the World Wide Web DNS does that job and finds that there is indeed an http:// capable endpoint living at address 94.76.200.117 
  3. the document /author.html is requested of that endpoint 
  4. the document representation is returned to the browser, note that /author.html is just the identifier of the document, how the response is created is unknown (nor should you care, that's the whole point of the World Wide Web)
Spaces in Resource Oriented Computing work in exactly the same way : 
  1. requests are issued into them
  2. resolution takes place to find an endpoint capable of handling the request
  3. the resource is requested of that endpoint
  4. the resource representation is returned
You might think ja, ja you've already said that last time. And we've seen it when using the Request Resolution Trace Tool. Where's the rest of it ? There is no rest. This is it !

There is REST though, but that's a topic for another post.

The main difference with the World Wide Web is that Resource Oriented Computing doesn't stop at just one space. Let us look at my own module template to see what that means :

<module version="2.0">
    <meta>
        <identity>
            <uri>urn:org:elbeesee:experiment:introduction</uri>
            <version>1.0.0</version>
        </identity>
        <info>
            <name>elbeesee / experiment / introduction</name>
            <description>elbeesee / experiment / introduction</description>
        </info>
    </meta>

    <system>
        <dynamic/>
    </system>


    <rootspace
        name="elbeesee experiment introduction"
        public="true"
        uri="urn:org:elbeesee:experiment:introduction">
                       
        <import>
            <uri>urn:org:elbeesee:experiment:introduction:import</uri>
            <private/>
        </import>
    </rootspace>


    <rootspace
        name="elbeesee experiment introduction import"
        public="false"
        uri="urn:org:elbeesee:experiment:introduction:import">
    </rootspace>
</module>


Right, a module with two spaces. Let us start with the generalities (and a couple of best practices) :

  • A space is identified by the uri attribute on the <rootspace> tag. If you do not specify one, one is generated for you. Specify one !
  • The attribute public on the <rootspace> tag specifies whether or not a space can be used (imported) outside of it's module. The default is true, however, specify it !
You may think I have an obsession with being explicit. I don't. I am however much in favour of the KWYAD approach. For the youngsters, that approach predates Xtreme programming, Agile, ... you name it ... and is still as valid today as it was 20 years ago. Know What You Are Doing.

At the bottom there is the urn:org:elbeesee:experiment:introduction:import space. It will contain resources that are only needed in the module itself (public="false"). These will typically be imports from other spaces.

At the top there is the urn:org:elbeesee:experiment:introduction space. It will contain the resources that we want to expose (public="true"). Note that it has the same uri as the module itself. That is a common practice for smaller modules that only expose a single space. That's not mandatory, if you think urn:org:elbeesee:experiment:introduction:public (or whatever) is clearer, by all means go with that.


This sometimes leads to confusion. Remember that modules have no relevance, but spaces do ! You can not import a module. You import a space.

The top space imports the bottom space. And marks the import as private. This means that none of the resources from the bottom space will get exposed. This is KWYAD SOP.

To experiment with this a bit, add an import of urn:org:netkernel:ext:layer1 in the bottom space :
    <rootspace
        name="elbeesee experiment introduction import"
        public="false"
        uri="urn:org:elbeesee:experiment:introduction:import">
       
        <import>
            <!-- data:/ scheme -->
            <uri>urn:org:netkernel:ext:layer1</uri>
        </import>
    </rootspace>


Right, before I can ask you to fire-up the Request Resolution Trace Tool ... I promised not to make - too many - assumptions so a bit of explaining is in order here. You just modified the module.xml file of the module we deployed earlier. NetKernel has picked up those changes. It has. That's what the <dynamic/> tag in module.xml does. While you can debate if that tag should be there in a production system, it is of huge value during development. You make your changes to module.xml and lo and behold, they are deployed.


Another assumption I'm making is that you know what Layer1 is. For now you'll have to settle with the explanation that it is a space with tools that you can freely reuse in your own modules. 


Ok, now set up the Request Resolution Trace Tool as you can see below (note that I selected the bottom space) :

Amongst - a lot of - other things, the Layer1 space contains the data: scheme which we're going to use for our tests. And yes, that is why I specified that as a comment. When I look back to this module in a couple of months, that comment will remind me of why I imported Layer1 (yep, KWYAD again).

The identifier we are doing to use is data:text/plain,Hello World. I bet you can deduce what that is going to do. Anyway, let us Resolve it :


What you see is that we go down into the imported space and get a hit. 

Now, urn:org:elbeesee:experiment:introduction:import is itself imported into urn:org:elbeesee:experiment:introduction. So is our identifier resolvable there ?

No it isn't. At least it isn't in the sense that it does not get exposed. You may already have noticed however that there seems to be a third space you can select in the drop-down. Select it and try to resolve again :

This isn't actually a third space, it is the private part (for internal use only) of the urn:org:elbeesee:experiment:introduction. To prove that the data: scheme is available internally, add the following snippet in there:

        <mapper>
          <config>
              <endpoint>
                  <grammar>res:/introduction/helloworld-data</grammar>
                  <request>
                      <identifier>data:text/plain,Hello World</identifier>
                  </request>
              </endpoint>
          </config>
          <space></space>
        </mapper>
 


Now try to Resolve and Inject identifier res:/introduction/helloworld-data in urn:org:elbeesee:experiment:introduction :

I'll explain the mapper endpoint in detail in a future post, as you can see however, there's quite a couple of Hello World examples possible.

Ok, that's a lot of information to digest. Before I sign off I want to make the point that the examples are far from childish. The whole World Wide Web is contained in one single space. The above explores three spaces that have at least the same potential. Think about it.

Yes, I know I promised to show you how to make your resources available in the browser. That is part of the space-story but although doing it is simple, the explanation is quite long. So I'm moving that bit to next week's post.

2013/10/25

back to the beginning ... hello world

FYI : I recently did a minimalistic refresh of my personal website. This may have broken a couple of links in this blog and elsewhere. No worries, in the coming weeks (starting today), I'm going to review a lot of basic stuff and all of the code examples are going to end up on Github.


It is frequently necessary to take a break, stand still (very important and not only for my three year old son) and look back from where you came. If you don't do this, assumptions you make will soon no longer be valid.

Don't do this when you've got a hungry tiger or bear chasing you. Erring on the side of caution is perfectly acceptable and can safe your life.

Recently I've been trying to teach myself at least one of the Functional Programming languages. Almost every book assumes - without explicitly saying so - that you like the language before you read the book. No, I don't. Almost every book on the subject assumes - without explicitly saying so - that you just see how much simpler functional code is as opposed to the equivalent in imperative code. No, I don't.

I said almost every book because I think I found one that doesn't make those assumptions and is willing to take my hand and show me. I'm working through it at the moment, I'll definitely let you know when I'm done.


By now you should be thinking : this is a clear case of the pot calling the kettle black. And in case you were not, I was myself. So starting today I'm going to go through a couple of the basics of NetKernel again. I will also try make any assumption explicit. Are you ready (forgive me the long introduction) ? Today we're revisiting Hello World.

Assumptions
  • You want to take a look at NetKernel. 
  • You have succeeded in installing NetKernel in a directory that I'll from here on describe as [installationdirectory]. 
  • You have an instance of NetKernel running and when you point your browser to http://localhost:1060, it shows the NetKernel Backend Fulcrum.
  • You are able to extract a directory from a zip archive.
I am very aware of the fact that the second and third assumption may already be a bit too much for a mere beginner, I am working on something to address that issue. Watch this space.

Source
In NetKernel there is a concept of a module, which you can describe as the unit of deployment or (with less pomp) as an application. The source of the application that we are going to discuss can be found here on Github. It's the urn.org.elbeesee.experiment.introduction directory (with content) that we are going to need on the system where you've got NetKernel running.


Again a lot of assumptions. If you're not familiar with git and Github, this link should get you a zip archive that contains the directory we need.

Deploying the module
If you already have some experience with creating your own modules, you probably have a location on the filesystem where you like those modules to reside. Otherwise :
  • Create a subdirectory project-modules underneath [installationdirectory].
  • Put the directory urn.org.elbeesee.experiment.introduction (with content) in the project-modules subdirectory.
  • If it doesn't exist yet, create a subdirectory modules.d underneath [installationdirectory]/etc.
  • Create a file elbeesee-experiment.xml in the modules.d directory with the following content :
    <modules>
      <module runlevel="7">project-modules/urn.org.elbeesee.experiment.introduction/</module>
    </modules>
Congratulations, you've just deployed a module !

Exploring the module
For this we are going to use the Request Resolution Trace Tool. Set it up (select the module and the space) as shown below.


What we're going to do is run through a couple of Identifiers and then Resolve, Inject and Display Representation for those. Every time we do that, we're going to explore our module's source to see if we can find why we got the result that we did. Let us start with Identifier res:/introduction/helloworld-file. A Resolve will look like this


This is the time to also start following along in the module source. Fire up your favorite plain text editor (anything but Notepad will do fine, I use both Eclipse and Sublime Text) and open up the module.xml file in the urn.org.elbeesee.experiment.introduction directory.

I am assuming a bit of familiarity with XML here.

In NetKernel, everything is a resource. You can think of a resource as an addressable bit of information. Resources live in spaces. You can think of a space as a mathematical set or a scope or a context. A resource can be requested with its space as the context. And that's it, there's no more. A module is no more than a carrier/container for a couple of (root)spaces that belong together but has no further meaning within NetKernel.

In the Request Resolution Trace Tool, we selected a space. You can find this same space in the module.xml file. Right at the top of that space you find this snippet of XML :

       <fileset>
            <regex>res:/introduction/helloworld-file</regex>
            <rewrite>res:/resources/txt/helloworld.txt</rewrite>
       </fileset>


Can you see the Identifier we used in the Request Resolution Trace Tool ? Can you deduct what this snippet is going to do ? Note that fileset endpoint (specific type of resource) assumes the files to be relative to the module.

The Resolve just checked if the Identifier was available in the space, lets try to Inject next.


Your output will very likely differ from mine in that the location of the module will differ (and you might be using a Linux OS ... und so weiter). The Inject actually requested the resource.

Now, when you request a resource, you never get the resource as a result. I'll leave philosophical debates about whether or not there actually is anything at all to (meta) physicists (they can't show you an electron any more or less than they can show you love or an idea or ...), the fact is that what you get from a request is a representation of a resource

This is not bullshit. If you accept that when you browse to http://google.com, what you get is not the http://google.com, but a representation of it and that at that same instant thousands of other people also get a representation of http://google.com ... you've understood the concept.

So, what the Request Resolution Trace Tool is telling us, is that the representation we got back from our resource request has type readable binary stream. Which is the standard representation type for a file.

But what does the readable binary stream contain ? Well, with Display Representation we can try to visualize it.

This must have felt like a huge roundabout way to show you the contents of the helloworld.txt file, but we've actually covered most of what Resource Oriented Computing is about while doing so !
 
Let us move on to Identifier res:/introduction/helloworld-literal. The Resolve will look like this

Interesting. Note how the fileset endpoint is checked first. Within a space, resolution is top down. The hit this time fell on this snippet of XML

        <literal type="string" uri="res:/introduction/helloworld-literal">Hello World</literal>

This is the NetKernel entry (hasn't been added yet) for the list of Hello World programs. Defining a resource doesn't get any easier than that.

When we Inject the request, the result looks like this
You are correct to say there's a fundamental difference here. We can already see the result ! The reason is that our representation is a java.lang.String. The toString() function applied to a String shows ... you guessed it ... the String, whereas the same function applied to a Readable Binary Stream does not show the contents of the Stream.

There ... I'm leaving the the final Identifier res:/introduction/helloworld-accessor as an exercise for you. It may look harder than the two others, but it is exactly the same. An accessor is a code based (in this case Java) resource request.

Also try a couple of Identifiers that can not be resolved, like res:/introduction/helloworld-yourown for example.

That's it for this episode. Next time I'm going to show how you can make these resources directly accessible from your browser so you don't have to constantly use the Request Resolution Trace Tool.