Arduino success

November 7, 2015

So, I picked up a project on the Arduino “Gigs and Collaborations” board. What the guy wanted seemed pretty straightforward, a half-an-hour job. I said I’d do it for fifty bucks.

You poor fool.

Over the week, our correspondence became increasingly heated.

To be fair, much of it was my fault. I simply hadn’t read a lot of what he wrote, but (again, too be fair) it was written in somewhat original syntax and was really a lot of work to decode. I was fooled by his address and his european-looking nom de internet. I have no doubt that this gentleman spent his formative years on the subcontinent or in regions further east.

Anyway. On the 5th, I sent a message with an attached sketch (arduino projects are called ‘sketches’), the message describing what the attached code does now. His reply to me contained – among other things – this beauty:

CAMERA NOTE: WHETHER THE CAMERA IS POWERED ALL THE TIME, OR JUST FOR 12 THE SECONDS I ASK TO HAPPEN WITH THE CALL TO “ACTION”. WHEN YOU HIT THE REC BUTTON IT RECORDS FOR 12 SECONDS AND STOPS IT CAN ONLY RECORD A 12 SECOND CLIP. IN OTHER WORDS,, YOU HAVE TO PUSH (TRIGGER) THE RECORD BUTTON AGAIN TO MAKE IT RECORD AGAIN. I USE THE 12 SECOND POWER VIDEO (THE “ACTION”) TO SAVE POWER. SO THE CAMERA DOESN’T RUN ALL NIGHT LONG. JUST WHEN IT IS CALLED. I DON’T SEE A NEED FOR “WAIT FOR MOTION SENSOR TO GO LOW?? BECAUSE IT “WILL”, MIGHT TAKE A MINUTE OR 2 OR 3, AND THAT DOESN’T MATTER. BUT IT WILL GO LOW.AFTER IS DOES, SIMPLY…PLAY ACTION!

Which frankly, I have only read in full just now. Now that I have completed the project, with hindsight I can see what the bloke is saying. (He’s wrong, by the way. Without a wait, the logic would trigger the start-up sequence over and over).

So what did I do? I’ll tell you what I did, man. I wrote a long screed with some word in CAPS saying “you said this, then; and you are saying this now”. Then I deleted it without sending it. With age, comes wisdom. Instead, I sent him this:

Ok, well the sketch I sent you last time should do what you want, then.

Have you tried it out?

Well, the situation is that he can only really try it out on the weekend. Dude has a job.

Today, I got this:

good morning Paul, it appears to work exactly as anticipated. I am very pleased, played with it all day yesterday. money will be in your account by the end of the day.

please keep my contact info. if you need a reference give them my email, I will speak highly of you. as with most of my prototypes, they seldom work perfectly the first time out, or, I discover “OH!” I want it to do this too!

so there may be some tweeking of this one needed down the road. if not this one, I have other projects for you to work on after this one is finished.

great job!

Fucking “Booyah!” and fist-pump.

Dude may be just a tiny bit optimistic about the possibility that I might accept further work from him. But this was definitely a success.

Also learned quite a bit about accepting work over the internet and corresponding with clients.


Dev Environment

June 23, 2015

So. To run our app on my local machine I need

  1. A database. Postgres.
  2. An LDAP server. Apache.
  3. The “mapper” webapp. Grails on 7070.
  4. The “services” webapp. Grails on 8080.
  5. The “editor” rebapp. Rails on 3000.

And then it all goes.

Or it did, until I started putting security annotations on the groovy services.

I’m guessing the problem is that the front end – the browser client – needs to pass login cookies to the server app in order to make certain JSON requests, and it won’t do this because the login cookie has a different origin.

So

  1. Reverse proxy. Squid on (say) 9090.

With a bit of luck, I can explain to squid to send editor requests to the editor and service requests to the services. Hopefully, the web browser will be completely fooled by this and merrily send along the authentication tokens.

I had hoped to avoid needing to know much about Shiro, spring security, and related issues. But, it seems there’s little avoiding it. :(

Man, I’m looking forward to getting my Apple trashcan and not having to wait a minute and a half to bounce everything. Not so much looking forward to putting aaaaall the stuff on it that I am going to need. Dear God I hope it just installs the stuff without undue fuss – gigabytes of crap, starting with XCode (which you need for the C compiler and other basic utilities). I’m going to have to fast-forward 5 years of software installation history at the site I work at, as well as moving to homebrew in place of macports. Maybe I should live-blog it. It’s bound to be hilarious, and involve a huge amount of very bad language.

Must remember not to do it on Wednesday – that’s when the clients drop by to discuss progress.


Git squash

June 15, 2015

Just for my own reference:

I do a lot of work on branches, and I like to check them in periodically as I achieve chunks of work. But it’s quite granular – way too granular for the main branch. So I’d like to compress them down into a single revision for whatever feature I am implementing for the main branch.

Here’s how it’s done. Let’s say I’m on branch NSL-1168 (I name my branches for their JIRA id). Everything needs to be clean and checked in.

First, the state of master after the most recent merge of NSL-1168 into it is tagged (to begin using this process, I have to do that manually). Then:

Grab the comments on all my incremental changes and concatenate them:

echo 'NSL-1168 squash merge' `date` > ~/tmp/merge_comment
echo >> ~/tmp/merge_comment
git log --ancestry-path NSL-1168-last-merge..NSL-1168 >> ~/tmp/merge_comment

Merge master into branch. At this stage, work may need to be done to make the merge work. Hopefully not.

git checkout NSL-1168
git merge master

Squash merge the branch into master. This is the magic bit. We merge and commit with the big long comment detaling all the changes.

git checkout master
git merge --squash NSL-1168
git commit -F ~/tmp/merge_comment

Finally, tidy up. The squash merge doesn’t produce an explicit record that the master and my branch were synced, so I merge master into the branch again – a no-op – just to document inside git that the branches are the same at that point.

Finally, I move the ‘last-merge’ tag. All my incremental changes from this point will go into the master as a bundle next time I do this.

git checkout NSL-1168
git merge master
git tag -f NSL-1168-last-merge

This leaves the branch one commit ahead of the master, which is ok. We do not merge the branch into the master because not doing that is the whole point of the exercise. Ultimately, the master will look like the branch has never been merged into it. But that’s ok – the squash merges will do the job. This means also that if the JIRA branch is deleted from the repository, then all of those incremental changes will be unreferenced and will disappear. Again – this is ok.


SKOS and classifications

April 29, 2015

“Oh, and we’d really like SKOS as well”.

Riiiight.

We have names and references.
Names appear in references at “instances” of a name.
Some name instances are taxonomic concepts.

We have a classification made up of tree nodes.
A tree node has many subnodes.
A tree node may be used by many supernodes as a subtree.
A tree node (for classification trees) refers to a taxonomic concept instance.

Now here’s the thing.

The basic job of SKOS in our data is to say “this concept is narrower-than that concept”. X ∈ A → X ∈ B. Naively, we’d put this over the node ids and say job done.

But our tree nodes are most certainly not taxonomic concepts. They are placements of concepts in some classification. It’s the concepts that are concepts. So the skos output must declare that it is the taxon concepts that are narrower-than other taxon concepts.

The problem becomes that the data becomes a mishmash. If a concept is moved from one place to another, then our classification data (which holds the entire history of a tree) will declare in one node that A⊂B and in another that that A⊂C, where B and C are different taxa at the same level.

Not good.

The problem becomes clearer when you consider having two entirely separate and incompatible classifications. It makes sense to say A⊂B according to the people at NSW University, but A⊂C according to the Norwegian National Classification of Herring. These classifications are sources of assertions that you can choose to trust or to not trust.

The nature of our classification nodes becomes clear. Each node is a document – a source of triples – that asserts certain facts and that trusts the assertion of facts in the nodes below it.

All I need to do is to find a suitable ontology for expressing this. The W3C provenance ontology looks promising. It even has – Oh My Lord! I has these predicates

prov:wasRevisionOf prov:invalidatedAtTime prove:wasInvalidatedBy

Which match exactly certain columns in my data.

(edit: why is this important? It indicates that the provenance ontology and my data may very well be talking about the same kinds of thing. If your data has a slot for “number of wheels” and “tonnage” and so does mine, there’s a good chance we are both discussing vehicles.)

I’m pretty comfortable, at this point, deciding that each tree node in a classification is a prov:Entity object. In fact, it seems to be a prov:Bundle in the sense of its subnodes, and a prov:Collection in the sense of the reified subnode axioms, which are skos terms.

I’m probably over-engineering things again. But there you go.


PS: not all subnode relationships are skos broader/narrower term relationships. In particular, hybrids. A hybrid cucumber/pumkin is not simply a kind of cucumber in the same way that a broadbean is a type of legume. I cannot simply assert skos inclusion over a tree – the actual link types and node types matter.


Twitter

March 5, 2015

Our team’s twitter account is https://twitter.com/AuBiodiversity . Follow for updates in regard to … stuff. Biodiversity stuff.


JENA, D2R, TDB, Joseki – happy together at last

March 3, 2015

Well!

The magic formula tuned out to be:

  1. Run joseki
  2. Using the d2rq libraries
  3. And just the two joseki jars containing the actual server
  4. Use a new version of TDB, which uses the new ARQ

And she’s up. Dayum. I have the static vocabulary files, the preloaded old APNI, AFD, and 2011 Catalogue of Life, and a live link to our (test) database, and you can write SPARQL queries over the lot. No worries. A trifle slow if you do stuff that spans the data sources.

Now the boss wants me to write some user doco for it. Item 1 is why this stuff matters (it matters because it’s our general query service). So rather than explaining it all here in my blog, I should do it on confluence.


JAR file hell with Jena and D2RQ

February 24, 2015

I would very much like for D2RQ to work as a subgraph inside an existing joseki installation. But sweet baby Jesus I just can’t make the nine million JAR file libraries talk to each other.

Tried to put the D2RQ assembler into joseki. Won’t run without an ‘incubation’ library. Blows up with a class file version exception, which means that the jars were compiled with other versions. Which is nuts, because the version of joseki I am using – 3.3.4 – is the same as is internally inside d2rq.

Tried to put the D2RQ assembler into fuseki. Fuseki attempts to call “init()” on the assembler, which ain’t there. According to the d2rq project, there is no such method on the interface, so clearly d2r was compiled to a different specification of jena than was fuseki.

Tried to launch the joseki that is inside the d2r installation (which obviously works) as a joseki instance rather than a d2r instance. Nope. joseki.rdfserver isn’t there.

Tried to get the d2rq and joseki source so as to compile them together on the same machine. But the build file specifies java versions, the git project HEAD points to a dev branch, and the joseki project isn’t even git.

I am at the stage of hacking up the d2r server code itself – it has the joseki and the d2rq classes living with one another, I have the source and it all compiles and builds ok. Issue is that when it launches, the “go and do it” class creates the d2r graph as a top-level object and as a default graph (from a sparql point of view). This won’t do – I need a top-level graph that is a stuck-together frankenstein that has the d2r component as a mere subsection of what is going on. The “go and do it” returns D2RQModel rather than the interface Model. Happily, I can fix at least that and it still compiles. So maybe I can build the graph that I want internally. But this means learning the programatic interface to jena – I already have assemblers that are correct (it’s just that they won’t run withou colliding into class file version issues). Perhaps just find the source of joseki.rdfserver and copy/paste it into the d2r project? Maybe that’s got a magic “read an assembler and spark up a SPARQL service” method?

If anyone out there has managed to get the d2r assembler working inside fuseki or joseki, or for that matter any implementation of a sparql endpoint, I would be terribly grateful for some tips.