I could not afford to wait long with rolling out an Apt repository for NetKernel 6.1.1. I'm happy to announce the wait is now over !
As with RHEL, there are two repositories available. This is again due to the possible difference in startup-scripts, there is no difference whatsoever with regards to NetKernel itself.
So, if your Debian-flavoured distribution is using SysV-style startup-scripts (the scripts live in /etc/init.d) you want to pick the 14.04 repository below. If it is using systemd startup-scripts, pick the 16.04 repository below.
In case you're wondering what the 14.04 and 16.04 stand for ... those are the Ubuntu LTS distributions that I tested on (in fact, since 15.04 Ubuntu is using systemd). As you'll understand I don't have every Debian-flavoured distribution available for testing (nor would I want to), but if your system is using Apt, chances are good the below will work for you.
On 14.04
Create a new repository entry (or have your System Administrator do this) in /etc/apt/sources.list :
## NetKernel
deb http://tomgeudens.io:8400/Debian/ 14.04/
In order to be able to work with this repository, you will have to add the public key to apt :
wget http://tomgeudens.io:8400/Debian/keys/aptrepo_public.key
sudo apt-key add aptrepo_public.key
Next, you update your apt cache :
sudo apt-get update
If all goes well, you should be able to install NetKernel :
sudo apt-get install netkernel-se
Now you've got a completely functional NetKernel 6.1.1. SE on your system. It lives in /opt/netkernel and has it's own service user, netkernel. It's not running yet though, because this is where some configuration comes in (see my previous post).
When you're ready to roll, this is what remains to be done :
sudo update-rc.d netkernel enable
sudo service netkernel start
On 16.04
Create a new repository entry (or have your System Administrator do this) in /etc/apt/sources.list :
## NetKernel
deb http://tomgeudens.io:8400/Debian/ 16.04/
The other steps are the same and when you're ready to roll, this is what remains to be done :
sudo systemctl enable netkernel
sudo systemctl start netkernel
Remarks
- You'll notice that the installation already contains the latest updates to NetKernel 6.1.1. The idea is that my repositories will always (give or take a couple of days, I'm only human) contain a version that does not require updates from Apposite.
- Apt has recently raised the ante on security requirements for signing the repository. The above repositories are both compliant with those requirements (and should therefore not give warnings when you update the cache).
With NetKernel 6.1.1. now out in the wild, all reasons are of course valid to install it at your local shop and show it off. Now, I don't know about your daily job, but mine is in a bank/insurance environment (at the moment, I'm a freelance consultant) and they are not so happy with :
- downloading a jar
- copying the jar to a target system
- manually installing the jar
While these steps are trivial in themselves, they are often considered unprofessional/untrustworthy.
So, while solving this problem for myself I decided to solve it for everybody else too. I'm therefore proud to present the NetKernel Yum Repository today. At the moment it contains a NetKernel 6.1.1. SE rpm, an EE rpm will soon be added.
How does it work ?
On RHEL6 / CentOS 6
Create a new repository definition file (or have your System Administrators create this, strangely enough this will rarely be considered a problem), for example /etc/yum.repos.d/netkernel.repo with the following content :
[netkernel]
name=NetKernel Yum Repository
baseurl=http://tomgeudens.io:8400/RHEL/6/$basearch
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://tomgeudens.io:8400/RHEL/6/$basearch/yumrepo_public.key
enabled=1
Notice that both the rpms in the repository and the repository itself are signed, no messing around with security, I've got to convince the security team of a bank here !
Next you should be able to search for NetKernel and install it (this may require you to download the yumrepo_public.key, just say yes when asked for it) :
sudo yum search netkernel
======= N/S Matched: netkernel ========
netkernel-se.x86_64 : NetKernel Standard Edition
sudo yum install netkernel-se
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
netkernel-se x86_64 6.1.1-el6 netkernel 26 M
Transaction Summary
================================================================================
Install 1 Package(s)
Total download size: 26 M
Installed size: 39 M
Is this ok [y/N]: y
Downloading Packages:
netkernel-se-6.1.1-el6.x86_64.rpm | 26 MB 00:04
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : netkernel-se-6.1.1-el6.x86_64 1/1
Verifying : netkernel-se-6.1.1-el6.x86_64 1/1
Installed:
netkernel-se.x86_64 0:6.1.1-el6
Complete!
Now you've got a completely functional NetKernel 6.1.1. SE on your system. It lives in /opt/netkernel and has it's own service user, netkernel. It's not running yet though, because this is where some configuration comes in (pretty much as you would also not start an Apache server before configuring it) :
- At some shops logs are kept in a central place. The /opt/netkernel/log directory is empty at this point and can therefore easily be relinked to a central location.
- You might want to change the JVM settings in /opt/netkernel/bin/jvmsettings.cnf. Memory is cheap, so add lots ...
- NetKernel (as it is set up here) requires access to the Apposite repositories. This may require you to review and change the HTTP Proxy Settings in /opt/netkernel/etc/kernel.properties.
- ...
When you're ready to roll, this is what remains to be done :
sudo chkconfig netkernel on
sudo service netkernel start
On RHEL7 / CentOS 7
There are a few (small) differences on the more recent RedHat-flavoured systems. The idea of runlevels and such has practically disappeared (some traces are left for backwards compatibility). So the package is a bit different. Lets start with creating a new repository definition file :
[netkernel]
name=NetKernel Yum Repository
baseurl=http://tomgeudens.io:8400/RHEL/7/$basearch
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://tomgeudens.io:8400/RHEL/7/$basearch/yumrepo_public.key
enabled=1
The yum-commands to search and install are identical, you will notice that the rpm that is installed is netkernel-se-6.1.1-el7.x86_64.rpm.
The configuration is also identical, to get things started however these are the commands :
sudo systemctl enable netkernel
sudo systemctl start netkernel
Questions
- Does the package verify the Java 1.8.x requirement ?
No, it does not at the moment, I'm still figuring out how I can cater for both OpenJDK and Oracle (and possibly others) at the same time.
- I'm running on Debian/Ubuntu, do you have an Apt repository available ?
Not yet, but watch this space.
- I notice you only have a x86_64 package available, can you also provide a i386 package ?
Yes I can, that's on my todo-list.
- Look, my shop really does not allow me to punch through the firewall to the Apposite repository. Can you provide a package that contains a few extra modules (x, y and z) and does not phone home ?
Well, if you really need that, contact me with your requirements and I'll see what I can come up with :
tom<dot>geudens<at>hush<dot>ai
practical<dot>netkernel<at>gmail<dot>com
Today sees the release of NetKernel 6.1.1. For those of you who know what NetKernel is, this is a good time to revisit it, for those of you who don't know ... this is an excellent time to check it out !
Did you know ...
- That NetKernel is not just another Java Framework ?
Follow the above link to find out what NetKernel is !
- That NetKernel is not just another application server ?
Follow the above link find out what NetKernel is !
- That NetKernel is a great choice for RDF (and Open Data in general) processing and publishing ? There are several existing implementations already and several more in the works !
- That you can find some of my working examples on Github ?
When writing you occasionally make bold statements. Rarely does somebody call my bluff. But somebody did and therefore I have to show you ten significantly different hello world examples.
One
<literal type="string" uri="res:/tomgeudens/helloworld-literal">Hello World</literal>
As I've said before, we should have that one in the list of Hello World programs.
Two
<accessor>
<id>tomgeudens:helloworld:java:accessor</id>
<class>org.tomgeudens.helloworld.HelloWorldAccessor</class>
<grammar>res:/tomgeudens/helloworld-java</grammar>
</accessor>
For the code itself I refer to earlier blogposts.
Three
<accessor>
<id>tomgeudens:helloworld:groovy:accessor</id>
<prototype>GroovyPrototype</prototype>
<script>res:/resources/groovy/helloworld.groovy</script>
<grammar>res:/tomgeudens/helloworld-groovy</grammar>
</accessor>
Which requires the script and the language import
<literal type="string" uri="res:/resources/groovy/helloworld.groovy">
context.createResponseFrom("Hello World");
</literal>
<import>
<!-- contains GroovyPrototype -->
<uri>urn:org:netkernel:lang:groovy</uri>
</import>
Four
<mapper>
<config>
<endpoint>
<grammar>res:/tomgeudens/helloworld-data</grammar>
<request>
<identifier>data:text/plain,Hello World</identifier>
</request>
</endpoint>
</config>
<space>
<import>
<!-- contains data:/ scheme -->
<uri>urn:org:netkernel:ext:layer1</uri>
</import>
</space>
</mapper>
Five
<mapper>
<config>
<endpoint>
<grammar>res:/tomgeudens/helloworld-file</grammar>
<request>
<identifier>file:/c:/temp/helloworld.txt</identifier>
</request>
</endpoint>
</config>
<space>
<import>
<!-- contains file:/ scheme -->
<uri>urn:org:netkernel:ext:layer1</uri>
</import>
</space>
</mapper>
Of course you need to replace the identifier with an existing file of your own.
Six
<mapper>
<config>
<endpoint>
<grammar>res:/tomgeudens/helloworld-fileset</grammar>
<request>
<identifier>res:/resources/txt/helloworld.txt</identifier>
</request>
</endpoint>
</config>
<space>
<fileset>
<regex>res:/resources/txt/.*</regex>
</fileset>
</space>
</mapper>
Seven
<mapper>
<config>
<endpoint>
<grammar>res:/tomgeudens/helloworld-freemarker</grammar>
<request>
<identifier>active:freemarker</identifier>
<argument name="operator">data:text/plain,${one} ${two}</argument>
<argument name="one">data:text/plain,Hello</argument>
<argument name="two">data:text/plain,World</argument>
</request>
</endpoint>
</config>
<space>
<import>
<!-- contains active:freemarker -->
<uri>urn:org:netkernel:lang:freemarker</uri>
</import>
<import>
<!-- contains data:/ scheme -->
<uri>urn:org:netkernel:ext:layer1</uri>
</import>
</space>
</mapper>
Eight
<mapper>
<config>
<endpoint>
<grammar>res:/tomgeudens/helloworld-http</grammar>
<request>
<identifier>http://localhost:8080/tomgeudens/helloworld-literal</identifier>
</request>
</endpoint>
</config>
<space>
<import>
<!-- contains http:/ scheme -->
<uri>urn:org:netkernel:client:http</uri>
</import>
</space>
</mapper>
Which requires that the first example is exposed on the frontend fulcrum.
Nine
<mapper>
<config>
<endpoint>
<grammar>res:/tomgeudens/helloworld-xpath</grammar>
<request>
<identifier>active:xpath</identifier>
<argument name="operand">
<literal type="xml">
<document>Hello World</document>
</literal>
</argument>
<argument name="operator">
<literal type="string">string(/document)</literal>
</argument>
</request>
</endpoint>
</config>
<space>
<import>
<!-- contains active:xpath -->
<uri>urn:org:netkernel:xml:core</uri>
</import>
</space>
</mapper>
Ten
<mapper>
<config>
<endpoint>
<grammar>res:/tomgeudens/helloworld-dpml</grammar>
<request>
<identifier>active:dpml</identifier>
<argument name="operator">res:/resources/dpml/helloworld.dpml</argument>
</request>
</endpoint>
</config>
<space>
<literal type="xml" uri="res:/resources/dpml/helloworld.dpml">
<sequence>
<literal assignment="response" type="string">Hello World</literal>
</sequence>
</literal>
<import>
<!-- contains active:dpml -->
<uri>urn:org:netkernel:lang:dpml</uri>
</import>
</space>
</mapper>
And there you go, ten resource oriented hello world examples. There are many more possibilities but I think the above show both that there is a lot available and at the same time show that the patterns are always the same. Enjoy.
Even the most humble of modern laptops today has multiple cores at its disposal. When you work Resource Oriented you benefit from the fact that resource requests are automatically spread over the available cores. However within one (root) request you typically make subrequests sequentially. In most cases this is exactly what you want as one subrequest provides the input for the next ... and so on.
There are cases however where you can benefit from parallel processing. A webpage, for example, can be composed from several snippets which can be requested in parallel. In a previous post I discussed the XRL language :
<html xmlns:xrl="http://netkernel.org/xrl">
<xrl:include identifier="res:/elbeesee/demo/xrl/header" async="true"/>
<xrl:include identifier="res:/elbeesee/demo/xrl/body" async="true"/>
<xrl:include identifier="res:/elbeesee/demo/xrl/footer" async="true"/>
</html>
Another use case for parallel processing is batch processing. In my last post I developed an active:csvfreemarker component. It applies a freemarker template to every csv row in an input file and writes the result to an output file. It works. However, the files I want processed contain millions of rows and applying a freemarker template does take a bit of time. Can parallel processing help ? Yes it can ! Here's the revelant bit of code :
while(vCsvMap != null) {
int i = 0;
List<INKFAsyncRequestHandle> vHandles = new ArrayList<INKFAsyncRequestHandle>();
while( (vCsvMap != null) && (i < 8) ) {
INKFRequest freemarkerrequest = aContext.createRequest("active:freemarker");
freemarkerrequest.addArgument("operator", "res:/resources/freemarker/" + aTemplate + ".freemarker");
for (Map.Entry<String,String> vCsvEntry : vCsvMap.entrySet()) {
freemarkerrequest.addArgumentByValue(vCsvEntry.getKey().toUpperCase(), vCsvEntry.getValue());
}
freemarkerrequest.setRepresentationClass(String.class);
INKFAsyncRequestHandle vHandle = aContext.issueAsyncRequest(freemarkerrequest);
vHandles.add(vHandle);
vCsvMap = vInReader.read(vHeader);
i = i + 1;
}
for (int j=0; j<i; j++) {
INKFAsyncRequestHandle vHandle = vHandles.get(j);
String vOut = (String)vHandle.join();
vOutWriter.append(vOut).append("\n");
}
}
The freemarker requests are issued as async requests in groups of eight. Their results are then processed in order in the for-loop.
Why eight ? That number depends on several things. The number of cores available, the duration of each async request, ... You'll need to experiment a bit to see what fits your environment/requirements. So actually the number should not be hard-coded. Bad me.
It has been a while since my last post. But like my friend who blogs here - check it out, he has style - I just can't stop doing it.
In my last post I presented the KBOData site. For all its fancy features, the real work is to get all the raw data (1.12 Gigabytes worth of csv files) into the correct format for the - monthly - database load. The database is Stardog so the csv has to be transformed into one of the rdf formats. Turtle was selected.
For your information, the 1.12 Gigabytes of csv gets turned into 9.68 Gigabytes worth of triples.
Now there are a lot of tools available in NetKernel and we could definitely write our own csv-processor but there are good libraries available. I selected Super CSV and created a library module with it. A library module provides - in case that wasn't clear - functionality to other modules.
I'm not going to discuss the whole module (which you can find here, the module name is urn.org.elbeesee.supercsv), if you've followed the Back to the beginning series most of it should be familiar. I am going to discuss the new stuff though.
I removed the class file and the supercsv jar file before checking the module into Github (both to safe space on Github and to avoid errors due to a different environment). This means the module will not work as is, you'll need to compile it yourself.
One. The version in module.xml matches the version of the Super CSV jar file (2.2.0 at the time I write this). This is good practice when you wrap a 3rd party software (as we are doing here).
Two. The module contains a lib directory underneath the module's root. This is where we're going to put the 3rd party jars. In this case super-csv-2.2.0.jar which you can get from the Super CSV download.
Three. We add functionality. The active:csvfreemarker accessor takes a csv file as input, applies a freemarker template to each line and writes the output to a different file. It assumes the first row of the csv to contain the column headers.
We could export the Super CSV classes so that they can be used directly in other modules. While there may be cases where this is useful, this often quickly leads to classloader hell. Keeping the 3rd party functionality wrapped inside is the best way to go.
Four. The accessor itself contains nothing special (you'll find it - minus the freemarker processing - in the examples on the Super CSV site).
Five. The active:csvfreemarker response just mentions the input file has been processed. It is the side-effect (the output file) that we are interested in, the response is expired immediately.
Six. A unittest is provided. You need to replace the values for infile, outfile and the stringEquals value with your own. The input file could for example contain this :
firstname,lastname
tom,geudens
peter,rodgers
tony,butterfield
tom,mueck
rené,luyckx
Which will result in this output file :
geudens tom
rodgers peter
butterfield tony
mueck tom
luyckx rené
Note that Freemarker allows a lot more in its templates than is shown in the unittest. Here's one from KBOData :
<#if CONTACTTYPE == "TEL">
<http://data.kbodata.be/organisation/${ENTITYNUMBER?replace(".", "_")}#id> <http://www.w3.org/2006/vcard/ns#hasTelephone>; "${VALUE}" .
<#elseif CONTACTTYPE == "EMAIL">
<http://data.kbodata.be/organisation/${ENTITYNUMBER?replace(".", "_")}#id> <http://www.w3.org/2006/vcard/ns#hasEmail>; "${VALUE}" .
<#elseif CONTACTTYPE == "WEB">
<http://data.kbodata.be/organisation/${ENTITYNUMBER?replace(".", "_")}#id> <http://www.w3.org/2006/vcard/ns#hasURL>; "${VALUE}" .
</#if>
Seven. Usage. In fact the unittest shows how to use the library module. You import the public space and provide freemarker templates as res:/resources/freemarker/[template].freemarker resources. Enjoy !
P.S. I noticed that so far I have assumed that you know how to set up your development environment in order to build a NetKernel module. If this is not the case, there are tutorials for Eclipse and IntelliJ in the documentation.
P.P.S. Applying a Freemarker request to every single line takes - even in NetKernel and even using the lifesaver for batch processing - a while (depending on the size of the input of course). In a next post I'll discuss how we can fan out the requests.
There's a scary thing out there. It can tell computers the chemical composition of your prescription drugs. It can tell computers who's the top scoring soccer player on the World Cup (once that competition gets underway of course). The computers can then act on that information. It is called The Web of Data and like it or not, it will change a lot of things in our everyday live.
While it uses the same technology as The Web, it is different in that it is not made for human consumption but for machine consumption. Everybody reading this blog knows what Wikipedia is, but did you know about DBPedia? The difference ? Well, a human can make sense of a regular webpage and can infer meaning from it. A machine can do no such thing. It needs the data in a structured format and it needs to be told what the meaning of the data is.
There is a steady movement towards making more and more data publicly available. Tim Berners Lee (yes, him again) described a five-star system for publishing data in this way. For once governments are leading the movement (often because there are regulations that make it mandatory for them to open up their data) although more and more corporations are joining every day.
So, when the Belgian government decided to publish the Belgian company data (KBO/BCE/CBE) as three-star data (csv files), Paul Hermans and myself decided to add a couple of stars. We created a KBOData showcase :
- Paul added a semantic layer, turning the csv files into rdf. NetKernel is used to do the actual transformation in batch.
- The resulting triples are stored into Stardog.
- Based on input from Paul, I developed a(n almost) general purpose module (very little customization needed) in NetKernel for publishing the data.
- NetKernel also takes care of the caching, both in-memory and persistent where needed.
- Benjamin Nowack added the user experience (also served on NetKernel), for while it is about machine consumption, a showcase implies there's something to see/do for humans too. Note that what you see as a human is exactly the same as what a machine 'sees', there is no second set of data.
We learned a lot during the process. For one we seriously underestimated the amount of data (more than 74 000 000 triples/facts). This will lead to more use of paging in a second iteration. NetKernel is a natural match for structured data with lots of transformations (which is what this is all about), but even NetKernel can not shield against an open-ended request for the whole database.
A bit of cutting-edge was added with the fragments-server. Linked Data Fragments is a recent development from Ghent university to make The Web of Data more scalable. So when I say paging, it is very likely that the whole site will be based on fragments in the next iteration.
If you're interested in the finer details and/or want a similar implementation for your data, contact Paul or myself and we'll help you along.
