When you give a young child a hammer, regardless of upbringing, culture, safety instructions, ... it'll start whacking everything with it. A hammer seems to be one of those universal tools that everybody can figure out.
When you give an IT professional NetKernel, regardless of upbringing, culture, safety instructions, ... he'll (not being sexist here, chances are - sadly enough - better than even that it'll be a he) start whacking every problem with it. NetKernel seems to be one of those universal tools that ... ok, the simile does break down a bit here ...
So I happen to know that several NetKernel gurus also use it for their home monitoring systems. These contraptions are called DeathStars.
This is an inside Resource Oriented Computing joke. After studying the capabilities of the DeathStar in Star Wars closely, Tony Butterfield concluded that it was impossible. The amount of computing power required that is. Unless they had ROC, right Tony ?
Recently I've been introduced to very small computers. Most people in the industry know about the Raspberry Pi and at some point it was demonstrated that NetKernel could run on a plug computer. More recently however, the market is being flooded with TV-sticks, usually running Android. Only, besides from plugging into the HDMI port of any modern screen or tv, these sticks come with dual and now even quad core ARM processors ! Surely we can think of something better than watching tv with that, no ?
That's José Jalapeño on a Stick. If you don't know who that is ... get a life !
So when RikoMagic listened to the masses and brought a stick on the market that natively runs Linux (Picuntu to be exact), I jumped and snapped up a couple (the MK802 III LE). They come pre-installed with a lot of packages, Java being one of them. How easy can you make it ?
Very easy. In fact, since this is my blog I can shamelessly put in some plugs. Andrew Kirby does the support for RikoMagic in the UK and if something (see further down for some examples) wasn't already worked out in detail, he worked it out for me. Immediately. There are many stick-providers out there, but RikoMagic has seriously gotten ahead in my book and I'm already drooling over the quad core LE (soon he tells me, soon ...).
Installing NetKernel ... booting NetKernel ... updating NetKernel ... piece of cake. I removed a couple of the bigger packages I didn't need (apache, mysql), but there's plenty of room on the stick really.
Couple of the downsides. The ARM Java doesn't give me a very high performance score (even though I have two cores). Also, the SD cards used are very sensitive to power interrupts, so make sure you've got a backup image (and know how to put one back).
No, I wouldn't start a Bitcoin mining operation with a box of these, but as a data collector and forwarder unit (aka DeathStar) it's perfect. Now, I'm not Joe/Harriet the Handyman/woman. My knowledge of home monitoring stuff is very limited. However, this blogpost about external temperature monitoring with Linux seemed like a good starting point.
I ordered the LinkUSBi and a couple of TSense sensors and waited for a couple of days for them to arrive. During those I read up on digitemp and I also installed a tool called owfs on my stick.
When ordering all this stuff, order as-locally-as-possible. Otherwise you're up for a surprise as you'll have to pay the customs fee too.
I connected the LinkUSBi to the stick, a TSense to the LinkUSBi. I check dmesg and lo and behold, the device shows up there. I look for the /dev/ttyUSB* device that should get created ... but it's not there. Damn. Let me spare you the frustration and the weeding out of outdated (or plain wrong) references on the internet. What I'm missing is ftdi_sio in the Linux kernel. Pretty standard in most kernels, but it's not on the stick.
Andrew (Andy) Kirby comes to my rescue. Note (plug again) that he doesn't have to, there's nothing wrong with the stick, works as designed. He puts a complete MK802 III LE kernel kit online for me with detailed instructions on how to build it. All I have to do is change one line in the kernel configuration and activate CONFIG_USB_SERIAL_FTDI_SIO. Next I flash it to the stick (again, detailed instructions provided by Andy).
With a simple command I can now see the external temperature. With a simple NetKernel module I execute that command every minute. A simple NKP connection forwards that information to an instance in the cloud. An instance that can receive temperature (and other of course) information from - potentially - thousands of such sticks.
Yes, I know, this is nothing world shaking. Deathstar 0.2 as I call this exercise is pretty useless in itself. That is however not the point. The point is that for very little money (they practically throw these TV-sticks at your head already and prices are still dropping, the sensors were likewise very cheap) you can put in place a complete botnet of collect-and-forward units. Can you think of an application ?
I didn't get it. No, really, I had seen the poink module in the apposite repository several times already, but I had dismissed it (forgive me, Thomas Hicks) as a variation on Ping Pong or something.
As you can read in some of my other recent posts, the business of massaging data into other formats (a process we call transreption in Resource Oriented Computing) is booming. A lot of data today lives in spreadsheets. Or lets call it by the brand, a lot of data today lives in excel sheets.
Natively that format is not available in NetKernel, but as my post of last week showed, that can be quickly mended. So I looked around for a likely candidate library and I found one, Apache POI.
I didn't see it. No, really, so I did a bit of studying of the library myself and came up with the stuff I needed for the task at hand. Next project comes round and again excel sheets are involved, so I smile happily, until my lead says ... right, I've tried PoiNK on it, but it doesn't quite seem to do the job, can you have a look at that. PoiNK ? Apache POI for NetKernel ! Oops.
Sorry about that Thomas. You're stuff is very good. It needed a bit of a refresh (work on it was done in 2008, the Apache POI library has evolved a bit since then), so I started on that. The result (representations and transreptors ready) is work in progress and can be found here.
Note the following things :
- Although I renamed things in my version, Thomas Hicks still retains the original rights to the code. At this time (2013/08/26) it is still his module in the Apposite repository and this is just an alternative that you can use if you run into a problem with the version in Apposite. Regardless of license (Apache 2.0 in this case), I always render onto Caesar what is Caesar's.
- I'm only provided sources. Watch this space for a couple of posts on building NetKernel modules soon.
- The Apache POI library has to be added in the lib-subdirectory. I tested it with just the poi-3.9-20131203.jar file and that works fine.
Enjoy !
It is rare in IT, but on occasion things do not have to be difficult or obscure :
String aMessage = aContext.source("arg:message",String.class);
ByteArrayOutputStream vBAOS = new ByteArrayOutputStream();
Document vMessageDocument = new Document();
PdfWriter.getInstance(vMessageDocument, vBAOS); vMessageDocument.setPageSize(PageSize.A8.rotate());
vMessageDocument.open();
Font vFont = new Font(FontFamily.COURIER, 14, Font.BOLD, BaseColor.BLACK);
vMessageDocument.add(new Paragraph(aMessage, vFont));
vMessageDocument.close();
ByteArrayRepresentation vBAR = new ByteArrayRepresentation(vBAOS);
INKFResponse vResponse = aContext.createResponseFrom(vBAR);
vResponse.setMimeType("application/pdf");
The relevant imports (in case you're conditioned to see Document as org.w3c.dom.Document) are :
import com.itextpdf.text.BaseColor;
import com.itextpdf.text.Document;
import com.itextpdf.text.Font;
import com.itextpdf.text.Font.FontFamily;
import com.itextpdf.text.PageSize;
import com.itextpdf.text.Paragraph;
import com.itextpdf.text.pdf.PdfWriter;
iText ® is a Belgian invention, check it out ! NetKernel ® is a British invention, check it out ! Integration of the former in the latter was quick and painless.
Have a great weekend !
DISCLAIMER 1.0 : This entry is related (everything in my life is related to working with ROC/NetKernel) to but not about working with ROC/NetKernel.
DISCLAIMER 2.0 : This has the makings of a rant. It is not, but if you are easily offended ... you've been warned.
Before I actually start foaming around the mouth, I'd like to set the stage. Back in 1995, I left mainframe development for the prestigious job of Database Administrator (how and why I left is a story in itself, but this is not the forum for that). IDMS. On mainframe.
Now, while being a commercial product (from Computer Associates back then, no idea who runs that show nowadays) an IDMS database was actually pretty simple (technically speaking) and to this very moment I can (in any programming language you care to mention) build a working single user version of it in less than a day.
One day I arrive at my desk to find a dusty tape. A note attached to it says : Tom, this is a database dump from five years ago. We can't restore it because it's a version of the database software we no longer have. However, there's some data on it that the Big Man has asked for. Can you get it ? P.S. Back then we applied encryption on the specific records you need (very sensitive information as you can imagine), we've attached a printout of the assembler routine we used for the encryption.
I love shit like that. Don't you ? That's the stuff that makes IT interesting. Finding a - still working - tape drive (everything was being moved to disks, I'm not that old) that could read the tape was a challenge in itself. Getting the files on disk another. Then reading up on the specifications of that old database software. Verifying what I read about it on the files that I had. Writing a database simulator that could read the files and get the records. Then learning about the assembler and writing a decryption routine. Took me week all in all, but I delivered the goods to the Big Man himself.
Why am I telling this old war story ?
- To show that I don't mind challenges. Quite the contrary.
- To show that I've done my bit of data cleaning (literally in some cases), massaging it into a - machine or human - readable form.
Nowadays I work a lot with/around Linked Data. And I really like the ideas behind it, where we build together towards a web of all things connected taking the combined knowledge of the human race to the next level. Without a sliver of sarcasm I applaud and support that !
Note that I highlighted together though. For I find that that bit is missing today. Linked Data today is the realm of a class of - quite aloof - priests. They talk in coded language which is often so obscure they themselves require quite a bit of decoding to get it right (if they agree on the meaning at all, which is funny, because that is exactly one of the things about linked data, the meaning has to be clear).
Now, most of the electronic data out there is legacy (in this case meaning non-linked-data). It will be so for quite a while to come because the growth of electronic data is going ever faster and only a relatively small portion of it is in a linked data form.
This means that there's going to be a lot of grunt work, a lot of work for simple people like myself, to massage such data into a linked data form. That in turn means we need to understand. Linked Data is not just the realm of the Eloi that let the machines sort out the rest, there are quite a lot of Morlocks here that work hard to keep the machines running you know !
So, I read there's four rules and five stars. Excellent. Here are my extras (instead of adding them you can of course also subtract stars if that works better for you) :
- The way you structure data is as simple as possible. Occam's razor does apply -> one additional star
- When you make your structure publicly available, you document it so an average human being can understand it -> one additional star
- When you make your structure publicly available, you provide examples, examples and some more examples ... and you keep those up to date with the latest version of your structure -> one additional star
- When you make your structure publicly available, you link (that's what it is about after all, isn't it), to (valid) alternatives, explaining why you chose to have your own structure -> one additional star
I would - personally - go so far as to say I'd rather not have your data and structure publicly available (which - if you've been subtracting stars - takes away the last star) otherwise, for the only thing that you are adding is complexity.
It has been a while since I last blogged, so ... hello and nice to see you again ! Yesterday I was allowed (and very privileged) to help Paul Hermans build a Linked Data Server for the Dutch government.
There's a more than natural match between NetKernel and Linked Data. Data transformations (aka transreptions) are bread-and-butter, the RESTOverlay makes mapping of the internet facing URLs pretty easy and dynamically building SPARQL queries is a breeze with a tool like the Text Recursion Language.
Now, I don't know if Paul noticed, but about ten minutes into our day, this error started popping up (in the console and cron log) :
HDS Space Aggregation Failure, response to [res:/etc/system/CronConfig.xml] of type [java.lang.String] from [urn:com:proxml:stelselvanbasisregistraties:server] not transreptable to HDS
Strange because there's no res:/etc/system/CronConfig.xml resource in the module. I let it rest as it didn't impact the actual functionality and I needed all my brain capacity to understand all the Linked Data requirements.
This morning however I did the actual test and used the Request Resolution Trace Tool to see what happened :
Oops ... it does resolve. Ok, what is going on here ?
- Dynamic Imports. A powerful capability of NetKernel. By adding a certain resource to your module (for example res:/etc/system/CronConfig.xml or res/etc/system/SimpleDynamicImportHook.xml), a relevant space of your module gets imported elsewhere. Of course, linked to this is a mechanism of finding these resources, which is called Space Aggregation.
- A wide RESTOverlay. Since the Linked Data server was to serve a whole domain, I used <basepath>/</basepath>.
- <auto404/>, another feature of the RESTOverlay, automagically returning a 404 resource if a request into the RESTOverlay fails.
While I had covered all bases for incoming requests through the Frontend Fulcrum, I hadn't even thought about internal requests that might get through to the - widely open - RESTOverlay.
The solution is very simple, add a Limiter endpoint before the RESTOverlay that catches all unwanted internal requests :
<endpoint>
<prototype>Limiter</prototype>
<grammar>res:/etc/system/
<regex type="anything"/>
</grammar>
</endpoint>
And this is where a convention actually comes to my rescue. In theory, there's no way of knowing what internal requests are going to try and spam my RESTOverlay. However, convention states that Dynamic Imports and such work with res:/etc/system/.* resources.
It going to be the hottest day of the year here today (35+ Celcius), so I'm off to a place with some shade now. Enjoy your weekend !
P.S. No, haven't forgotten about the CRUD server and all. Working on it, watch this space !
It is fair to say I'm a fan of NetKernel. And I know quite a bit of the background of both the main protagonists. Today I'm therefore a very happy puppy, because over on his blog, Tony Butterfield is blowing the trumpet on his latest creation - ROC Hockey.
This game gives a new dimension to the classic game of Air Hockey. But that's not the important issue. The issue is that Tony has come full circle. Back when I was cutting my first IT-teeth on the MSX, Tony was already an accomplished game designer/developer/... (everybody did everything in those days, specialization is for insects).
Don't take my word for it, find an Memotech MTX emulator and find the rom for PotHole Pete. Unless you've recently finished Braid in one sitting (I don't think so dude or dudette, I don't think so), I guarantee you you're going have to put grinding on WoW (no, doesn't deserve a link) aside for quite a bit to finish it.
Modest as he is, Tony is downplaying what is out there, instead highlighting the weak points of the game mechanics. It's a British thing, forgive him. He's going to move this forward, it's going to be playable on Facebook ! And more importantly, it is going to be the first of many more and better ROC-based games to come.
And let me tell you, you'll feel the adrenalin pumping when you've figured out the scoring system and you're catching up quickly and there's just one square left in the game and you're going to take that b*stard down ... down I said ... you hear me ...
Sorry, I got carried away a bit there. I'll let you go and check it out now. ROC Rules !
I've not been such a good boy recently in keeping this blog updated. I offer no apologies, only the promise that things should pick up now ...
