It was to be expected, the moment I try to take things from the top, there is a major update to the NetKernel administrative interface (aka Backend Fulcrum). If you haven't done the update yet, you should, it is astonishing. I personally have a blind spot in the user experience area (being a command line jockey) but even I can see this is a huge improvement.
No, I'm not going to update my prior posts with new screen shots. Things have been cleaned up and integration between the tools is now much tighter. However, there is no change to what lies underneath.
An overhaul of that magnitude would in most products warrant a major release change, however, the crew @ 1060 Research is too honest and rated the two-week effort with only a minor release.
In my previous three posts I looked at the core of Resource Oriented Computing. In this and couple more posts I am going to discuss a couple of very common patterns. Today the spotlight is on the mapper.
== Intermezzo (you can safely skip this) BEGIN ==
Before I embark on that quest - creating a real service as I do so - I want to spend a moment on what is known as the magic/silver bullet fallacy. Also known as the deus ex machina fallacy :
- There is no such thing as a free lunch (no, not even in the Google/Apple/... offices). You can safely ignore any paradigm that claims it will solve problems effortlessly. That goes against the second law of thermodynamics. ROC claims no such thing. What it does claim is that it allows you to solve problems a lot more efficiently !
- I can explain it to you but I can't understand it for you. I got this bon mot from the Skype status of a very smart developer. Ironically he didn't quite understand ROC, but that is not the point. The point is that effort is involved.
Why am I saying this ? Because the service I'm going to create in this blog involves the use of an XSLT transformation and stylesheet. Maybe you master this technology, maybe you don't. You don't have to master it to follow the example, but my point is that you need some tools in your IT toolbelt. NetKernel comes with a lot of tools in the box (and you can add your own too). It will take effort to master them.
== Intermezzo (time to pay attention again) END ==
Amazon Web Services has grown into a large platform with a plethora of services (ec2, s3, ...). You can integrate these into your own solutions using the SDK or an HTTP service. In the latter case, each service has its own URL, which also depends on the region that you want to use the service in.
I'm ignoring the SDK for now. Integrating 3rd party libraries is a topic for another post.
In order to keep track of all these URLs, Amazon has published a regions.xml resource. As you can see, it contains all the URLs for all the services in all the regions. Woot !
But what if I just want to know what the Hostname is for the S3 service in the eu-west-1 region ? Well, that's the service I am going to build. It will define this resource : res:/amazonaws/baseurl/{service}-{region}.
For a first iteration we are going to download the regions.xml file and include it into our module itself as a file resource, res:/resources/xml/regions.xml.
<rootspace
name="amazonaws sdk import"
public="false"
uri="urn:com:amazonaws:sdk:import">
<fileset>
<regex>res:/resources/xml/.*</regex>
</fileset>
</rootspace>
The regions.xml file is an XML document. Processing an XML document can be done in many ways, I like using XSLT which takes the data and applies a transformation stylesheet to it. So we are also going to require a stylesheet, regions.xsl, which is also going to be a file resource, res:/resources/xsl/regions.xsl. We could define another fileset for that, but with a small change to the above we can cater for both :
<rootspace
name="amazonaws sdk import"
public="false"
uri="urn:com:amazonaws:sdk:import">
<fileset>
<regex>res:/resources/(xml|xsl)/.*</regex>
</fileset>
</rootspace>
The stylesheet itself looks like this :
<xsl:stylesheet
version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:nk="http://1060.org"
exclude-result-prefixes="nk">
<xsl:output
method="text"
encoding="UTF-8"
media-type="text/plain"/>
<xsl:param name="service" nk:class="java.lang.String"/>
<xsl:param name="region" nk:class="java.lang.String"/>
<xsl:template match="/XML">
<xsl:if test="Regions/Region[Name=$region]/Endpoint[ServiceName=$service]/Hostname">
<xsl:value-of select="Regions/Region[Name=$region]/Endpoint[ServiceName=$service]/Hostname"/>
</xsl:if>
<xsl:if test="not(Regions/Region[Name=$region]/Endpoint[ServiceName=$service]/Hostname)">
<xsl:text>unknown</xsl:text>
</xsl:if>
</xsl:template>
</xsl:stylesheet>
There, that's that. NetKernel comes with a lot of tools for processing and transforming data and since its roots lie in the XML era, it is no surprise that an XSLT transformation engine is readily available. It is part of the xml:core toolset. Including xml:core changes the above space to this :
<rootspace
name="amazonaws sdk import"
public="false"
uri="urn:com:amazonaws:sdk:import">
<fileset>
<regex>res:/resources/(xml|xsl)/.*</regex>
</fileset>
<import>
<!-- active:xsltc -->
<uri>urn:org:netkernel:xml:core</uri>
</import>
</rootspace>
Right, as you probably deduced, I've put our bill-of-material in a private space, as I don't want any of it publicly available. Before we get to the public part, lets see for a moment what we can do in this space :
The full declarative request is :
<request>
<identifier>active:xsltc</identifier>
<argument name="operand">res:/resources/xml/regions.xml</argument>
<argument name="operator">res:/resources/xsl/regions.xsl</argument>
<argument name="region">data:text/plain,eu-west-1</argument>
<argument name="service" >data:text/plain,s3</argument>
<representation>java.lang.String</representation>
</request>
And no, I haven't done anything strange or new with that (although it is a new - and very welcome - feature of the Resource Trace Tool), it is the equivalent of :
active:xsltc+operand@res:/resources/xml/regions.xml+operator@res:/resources/xsl/regions.xsl+region@data:text/plain,eu-west-1+service@data:text/plain,s3
Which is certainly a level up from a simple resource request like res:/resources/xml/regions.xml (try that first if you're uncomfortable with the above) but you can see how it is actually merely pulling together four resource requests as input for the active:xsltc resource request. This process is called composing.
If you've looked at the documentation for active:xsltc, you're might wonder why I have the extra "region" and "service" arguments. Well, they are varargs (variable arguments) which in this case are "optional parameters provided to the stylesheet runtime". You'll find them used in the stylesheet.
What is left is to map our proposed resource res:/amazonaws/baseurl/{service}-{region} to the above request in a public space :
<rootspace
name="amazonaws sdk"
public="true"
uri="urn:com:amazonaws:sdk">
<mapper>
<config>
<endpoint>
<grammar>
<simple>res:/amazonaws/baseurl/{service}-{region}</simple>
</grammar>
<request>
<identifier>active:xsltc</identifier>
<argument name="operand">res:/resources/xml/regions.xml</argument>
<argument name="operator">res:/resources/xsl/regions.xsl</argument>
<argument method="as-string" name="region">[[arg:region]]</argument>
<argument method="as-string" name="service">[[arg:service]]</argument>
<representation>java.lang.String</representation>
</request>
</endpoint>
</config>
<space>
<import>
<uri>urn:com:amazonaws:sdk:import</uri>
<private/>
</import>
</space>
</mapper>
</rootspace>
While this may not all be immediately clear, it should be clear what happens. An incoming resource request of the form res:/amazonaws/baseurl/{service}-{region} gets mapped to an active:xsltc resource request which is issued into the wrapped space. The response representation goes the other way.
The method="as-string" attribute turns values into resources. Don't worry about that too much at this point. I could have specified those arguments as follows :
<argument name="region">data:text/plain,[[arg:region]]</argument>
<argument name="service">data:text/plain,[[arg:service]]</argument>
but then I'd have had to import the Layer1 space as I explained in my post about spaces.
Check out the result :
If you remember from last time (extra credits to you !) how to make this available in your browser as http://localhost:8080/amazonaws/baseurl/s3-eu-west-1 ... you've just created a real and useful service. Congratulations !
Having regions.xml as a file in your module is fine, but it would be better if we could use the regions.xml that Amazon has published. You can (this will require internet access to work). First replace
<argument name="operand">res:/resources/xml/regions.xml</argument>
with
<argument name="operand">https://raw.github.com/aws/aws-sdk-java/master/src/main/resources/etc/regions.xml</argument>
Then redeploy the module and check out the result :
The problem is that your space doesn't know yet how to resolve a http:// resource. Not a problem though, NetKernel has a toolkit available that does know :
<rootspace
name="amazonaws sdk import"
public="false"
uri="urn:com:amazonaws:sdk:import">
<fileset>
<regex>res:/resources/(xml|xsl)/.*</regex>
</fileset>
<import>
<!-- active:xsltc -->
<uri>urn:org:netkernel:xml:core</uri>
</import>
<import>
<!-- http:// scheme -->
<uri>urn:org:netkernel:client:http</uri>
</import>
</rootspace>
Redeploy, try again and you'll get the same result as before.
Summary
- We've looked at an interesting pattern today, the mapper pattern.
- We've composed resources together into new resources, we wrote no custom code.
- We've made use of tools provided in NetKernel.
- We have build a real service.
It is a lot of information and you may feel that I'm taking a huge leap forward from where I left off last time, but do take it one step at a time and you'll see that is not the case. Enjoy !
Today I'm making good on my promise. We're going to make the res:/introduction/helloworld-xxxxx resources available in your browser. If the host you are running NetKernel on is open to the Internet, we might even make them available to the whole world !
I've had some questions about it so I want to repeat one point I made. The last two posts did cover the core of Resource Oriented Computing ! There really is no more. You do have all the building blocks and not unlike those in Lego they are simple. There is however no - practical - limit to what you can build with them.
Before we start making changes to our introduction module, I first have to explain about transports. NetKernel provides a Resource Oriented ... abstraction or box or - my personal view as a former System Administrator - operating system and until now we were only able to request the resources inside that with the Request Resolution Trace Tool. I bet that is not how you want to run your software. Transports solve this problem. A transport listens for incoming requests and brings them inside. The response (resource representation) goes the other way.
That may sound complex but again you are most likely already familiar with this concept. If you're a database administrator, you might know that an Oracle server listens on port 1521 for incoming database requests. The Apache server listens on port 80 for incoming http:// requests. The ssh daemon listens on port 22 for incoming ssh requests. Und so weiter.
As you can see thinking of NetKernel as an operating system was not so silly, for indeed, the transports it comes with work in exactly the same way. NetKernel has several transports available, out-of-the-box it has two running. On port 8080 and port 1060 it has Jetty (alternative for Apache) server based transports listening for incoming http:// requests.
The transport on port 8080 is also known as the FEF - Front End Fulcrum, the one on port 1060 as the BEF - Back End Fulcrum (or Administrative Fulcrum). For me personally this last acronym is confusing as BEF also stands for Belgian Franc (obsolete since 2002), I guess my age is starting to show.
You may already be able to deduce where this is going, these transports translate incoming http:// requests into res:/ requests.
For example :
- You request http://localhost:1060/tools/requesttrace in your browser.
- The transport listening on port 1060 translates this into a res:/tools/requesttrace request.
- The res:/tools/requesttrace resource can be found in the urn:org:netkernel:ext:introspect space (no worries about how I found that out).
- Therefore it must follow that the space that contains the transport - that would be urn:org:netkernel:fulcrum:backend - has access to (imports) the urn:org:netkernel:ext:introspect space.
We are going to proof the above by making our res:/introduction/helloworld-xxxxx resources available to the FEF.
We could just as well do it on the BEF, go ahead and try that as an exercise if you like !
The FEF lives in [installationdirectory]/modules/urn.org.netkernel.fulcrum.frontend-1.5.12. Change the module.xml file in there to import the urn:org:elbeesee:experiment:introduction space. Where ? Well, right underneath the comment :
<!--Import application modules here-->
<import>
<uri>urn:org:elbeesee:experiment:introduction</uri>
</import>
Save and try http://localhost:8080/introduction/helloworld-file in your browser.
Something went wrong. If you've been very attentive you may have noticed that the module.xml file of the FEF does not contain a <dynamic/> tag, changes to the module.xml file are not picked up. Stop and start NetKernel and try again.
Success. We have proved the concept.
However, if you're anything like me, it doesn't feel like success. Sure, it works, but that's quite a process to go through. Changing a file and stopping and starting NetKernel to deploy a resource to the FEF ? Anno Domini 2013 ? Really ? In fact, it was the process to follow in NK3. Things have changed.
You may undo your changes to FEF module.xml, stop and start NetKernel again (the introduction urls shouldn't work any longer) and then I want to draw your attention to this snippet :
<!--Dynamic Imports-->
<endpoint>
<prototype>SimpleImportDiscovery</prototype>
<grammar>active:SimpleImportDiscovery</grammar>
<type>HTTPFulcrum</type>
</endpoint>
What does that mean ? Well, it means that the FEF will automatically import spaces that contain a certain resource. So instead of changing the FEF, we're going to change our own module and add the following to the
urn:org:elbeesee:experiment:introduction space :
<literal type="xml" uri="res:/etc/system/SimpleDynamicImportHook.xml">
<connection>
<type>HTTPFulcrum</type>
</connection>
</literal>
The uri and content of the resource are what the SimpleImportDiscovery endpoint expects. You can also provide the resource in any of the other forms that we have seen (a file resource is very common).
And you'll notice the introduction urls work once more.
Not only is this very convenient (and we take that for granted in this day and age), it is also a very powerful pattern that you can use for your own solutions !
This brings me to the end of this post. Before I go I'd like to make an important observation. It is very easy to forget - I know I did for quite a while - that other than the default (http://) transports can be used to trigger resource requests. There's a cron-transport. An ssh-transport. A mail-transport. And if all of these available transports don't fill your needs, it has been shown that any of the Apache Camel Components can be turned into a custom transport for NetKernel.
I got some very nice feedback on my previous post (thank you, you know who you are), so I'm starting this time with the assumption that there is demand for basic information. Today we are taking a closer look at spaces.
You are almost certainly familiar with the concept. If you are into mathematics, you might consider a space as a set. If you are into programming, you might consider a space as a scope.
The very fact that you are reading this post, means that you have
issued a http:// request into a space that knows how to resolve such
requests ... a space better known as the World Wide Web. You may or may
not understand in detail how it (the World Wide Web) works but you
certainly issued that request with a pretty good idea of what you were
going to get.
Let us look at one such request :
- http://netkernelbook.org/author.html is requested in the World Wide Web space, you can do that with a - almost - magic tool called browser.
- resolution takes place, in the
case of the World Wide Web DNS does that job and finds that there is
indeed an http:// capable endpoint living at address 94.76.200.117
- the document /author.html is requested of that endpoint
- the document representation is
returned to the browser, note that /author.html is just the identifier
of the document, how the response is created is unknown (nor should you
care, that's the whole point of the World Wide Web)
Spaces in Resource Oriented Computing work in exactly the same way :
- requests are issued into them
- resolution takes place to find an endpoint capable of handling the request
- the resource is requested of that endpoint
- the resource representation is returned
You might think ja, ja you've already said that last time. And we've seen it when using the Request Resolution Trace Tool. Where's the rest of it ? There is no rest. This is it !
There is REST though, but that's a topic for another post.
The main difference with the World Wide Web is that Resource Oriented Computing doesn't stop at just one space. Let us look at my own module template to see what that means :
<module version="2.0">
<meta>
<identity>
<uri>urn:org:elbeesee:experiment:introduction</uri>
<version>1.0.0</version>
</identity>
<info>
<name>elbeesee / experiment / introduction</name>
<description>elbeesee / experiment / introduction</description>
</info>
</meta>
<system>
<dynamic/>
</system>
<rootspace
name="elbeesee experiment introduction"
public="true"
uri="urn:org:elbeesee:experiment:introduction">
<import>
<uri>urn:org:elbeesee:experiment:introduction:import</uri>
<private/>
</import>
</rootspace>
<rootspace
name="elbeesee experiment introduction import"
public="false"
uri="urn:org:elbeesee:experiment:introduction:import">
</rootspace>
</module>
Right, a module with two spaces. Let us start with the generalities (and a couple of best practices) :
- A space is identified by the uri attribute on the <rootspace> tag. If you do not specify one, one is generated for you. Specify one !
- The attribute public on the
<rootspace> tag specifies whether or not a space can be used
(imported) outside of it's module. The default is true, however, specify it !
You may think I have an obsession with being explicit. I don't. I am however much in favour of the KWYAD approach. For the youngsters, that approach predates Xtreme programming, Agile, ... you name it ... and is still as valid today as it was 20 years ago. Know What You Are Doing.
At the bottom there is the urn:org:elbeesee:experiment:introduction:import space. It will contain resources that are only needed in the module itself (public="false"). These will typically be imports from other spaces.
At the top there is the urn:org:elbeesee:experiment:introduction space. It will contain the resources that we want to expose (public="true"). Note that it has the same uri as the module itself. That is a common practice for smaller modules that only expose a single space. That's not mandatory, if you think urn:org:elbeesee:experiment:introduction:public (or whatever) is clearer, by all means go with that.
This sometimes leads to confusion. Remember that modules have no relevance, but spaces do ! You can not import a module. You import a space.
The top space imports the bottom space. And marks the import as private. This means that none of the resources from the bottom space will get exposed. This is KWYAD SOP.
To experiment with this a bit, add an import of urn:org:netkernel:ext:layer1 in the bottom space :
<rootspace
name="elbeesee experiment introduction import"
public="false"
uri="urn:org:elbeesee:experiment:introduction:import">
<import>
<!-- data:/ scheme -->
<uri>urn:org:netkernel:ext:layer1</uri>
</import>
</rootspace>
Right, before I can ask you to fire-up the Request Resolution Trace Tool ... I promised not to make - too many - assumptions so a bit of explaining is in order here. You just modified the module.xml file of the module we deployed earlier. NetKernel has picked up those changes. It has. That's what the <dynamic/> tag in module.xml does. While you can debate if that tag should be there in a production system, it is of huge value during development. You make your changes to module.xml and lo and behold, they are deployed.
Another assumption I'm making is that you know what Layer1 is. For now you'll have to settle with the explanation that it is a space with tools that you can freely reuse in your own modules.
Ok, now set up the Request Resolution Trace Tool as you can see below (note that I selected the bottom space) :
Amongst - a lot of - other things, the Layer1 space contains the data:
scheme which we're going to use for our tests. And yes, that is why I
specified that as a comment. When I look back to this module in a couple
of months, that comment will remind me of why I imported Layer1 (yep, KWYAD again).
The identifier we are doing to use is data:text/plain,Hello World. I bet you can deduce what that is going to do. Anyway, let us Resolve it :
What you see is that we go down into the imported space and get a hit.
Now, urn:org:elbeesee:experiment:introduction:import is itself imported into urn:org:elbeesee:experiment:introduction. So is our identifier resolvable there ?
No it isn't. At least it isn't in the sense that it does not get exposed. You may already have noticed however that there seems to be a third space you can select in the drop-down. Select it and try to resolve again :
This isn't actually a third space, it is the private part (for internal use only) of the urn:org:elbeesee:experiment:introduction. To prove that the data: scheme is available internally, add the following snippet in there:
<mapper>
<config>
<endpoint>
<grammar>res:/introduction/helloworld-data</grammar>
<request>
<identifier>data:text/plain,Hello World</identifier>
</request>
</endpoint>
</config>
<space></space>
</mapper>
Now try to Resolve and Inject identifier res:/introduction/helloworld-data in urn:org:elbeesee:experiment:introduction :
I'll explain the mapper endpoint in detail in a future post, as you can see however, there's quite a couple of Hello World examples possible.
Ok, that's a lot of information to digest. Before I sign off I want to make the point that the examples are far from childish. The whole World Wide Web is contained in one single space. The above explores three spaces that have at least the same potential. Think about it.
Yes, I know I promised to show you how to make your resources available in the browser. That is part of the space-story but although doing it is simple, the explanation is quite long. So I'm moving that bit to next week's post.
