Not quite as naive about privacy as before

A while back I wrote an article on how were we being naive about privacy in that we're all talking about the subject but no-one seems to be actually asking the question of what it is.

In order to answer this we've* taken an ontological approach by decomposing concepts into things such as information type, security class, jurisdiction, purpose, usage, provenance etc. All those concepts which make sense to the engineers who have to build information systems.

*Ora Lassila (he of RDF fame) has had a pretty big (huge!) hand in all of this too. Hey! We even got demonstration and prototype implementations working!

No work like this is even done in isolation - ontological approaches aren't new and certainly security, privacy, risk management etc have been tackled in one way or another - Solove, Schneier just to name two big names and a host of other researchers along too.

Now this is where I have a lot of hope: there is quite a bit a work in this area - that is, formalising concepts of privacy and in particular risk and risk avoidance in this ontological manner. There's even work on matching ontologies together. We start to see the real, fundamental structure of privacy and its conceptual basis.

What this means in the long term (and even the short!) is that we have a common terminological and semantic framework from lawyers to programmers coming into place.

We're missing some parts of course: how do all these ontologies fit together?  Can we unify the notions of consent used by lawyers with the [boolean] data types used by programmers?

"Your privacy is important to us"

bool optedIn = False  //sensible default

Actually we do in part - myself and Ora did develop quite a nice unification framework to link the ontologies together, link with the idea of information, link it with the notions of database table, CSV structures, classes etc; and even link it with how systems such as HADOOP process data.

So this gets me to a few places:

1. There is work on this being made - various groups are developing ontologies to express relevant concepts about information and aspects of information
2. Some groups are unifying those and drawing out subtle semantic differences
3. Some groups are applying these to more abstract areas such as the notions of consent and notice and how these may be made more meaningful to machines, and I hope humans too

We're starting to fill in the technical, fundamental details of what privacy is and now I find myself working with how we link these structures with the ideas of data flow modelling, risk management and software lifecycle requirements which requires all of the above.

References

Cena, Dokoohaki, Matskin. (2011) Forging Trust and Privacy with User Modeling Frameworks: An Ontological Analysis. STICS2011 (Draft)

Anya Kim and Jim Luo and Myong Kang (2005) Security Ontology for Annotating Resources. Research Lab, NRL Memorandum Report, pp51.

Kost, Freytag, Kargl, Kung. Privacy Verification using Ontologies

Golnaz Elahi, Eric Yu, Nicola Zannone (2009) A Modeling Ontology for Integrating Vulnerabilities into Security Requirements Conceptual Foundations?

Security and Privacy Ontology

A Particle Physics Approach to Privacy

A while ago I read on the LtU programming language blog a discussion about the future of programming language research - is the going to be any? Haven't we metaphorically illuminated all there is to see about programming languages?  Surely the future discoveries and developments in this area are going to be very small and very deep?

One reply caught my eye:

"When you've searched around the lamp, trying looking underneath it."

And so I feel the same about privacy...we're spending huge amounts of time looking at its effects and very little and looking at what it is.

What are the fundamental structures of privacy and how to these manifest themselves upon the World at large?

Should we take a highly deconstructive approach to privacy? Break it apart into its constituent blocks, its fundamental atomic and sub-atomic structure?

In the same way as the LHC breaks apart subatomic particles to reveal the inner structure of the Universe, should we take a similar approach to privacy?

What are the subatomic particles, the quarks, the bosons, the fermions of privacy? Does it have a metaphorical Higgs-boson and related field which gives privacy its "mass"?

Structuring Privacy Requirements pt 1

One of the toughest problems I'm having to solve, not just for my book on privacy engineering, but in my daily job as well is to formulate a set of privacy requirements to the software engineers and the R&D teams.

Actually it isn't that the privacy requirements themselves are hard, we've plenty at the policy level and extrapolating these down into the more functional and security related requirements at the implementation level (ok, it is difficult but there are harder things in this area).

Placing all these in a structure that ties together the various classifications and aspects of information, data flow modelling, requirements and risk management has been interesting and fiendishly difficult to say the least. Getting this structure into a state that supports all of these and getting the semantics of the kinds of thing it is structuring is critical to understanding how all of this privacy engineering works.

We assume that we understand the classification systems, for example the almost traditional Secret-Confidential-Public style security classifications and the Location, Time etc classifications of information type; as well as the other aspects such as purpose, usage, provenance, identity etc. Each of these has its own set of classification elements, hierarchy and other ontological structure. For example for information type:

Information Type Hierarchy

We also assume we understand data flow modelling with its processes, flows, sources, sinks and logical partitionings. We can also already see the link between elements here (as shown below) and the classification systems above

Example Data Flow with Annotations from Previously Described Ontologies/Classification Systems

Now the structure of our requirements needs to take into consideration the various elements from the classification systems, the aspect of the requirement we want to describe (more on this below) and the required detail of requirement relevant to the stage in the software process. This gives us the structure below:

So if we wish to find the requirements for User's Home Address Policy for Local Storage then we take the point corresponding to those coordinates. If there happens to be nothing there then we can use the classification systems' hierarchies to look for the requirement corresponding to a parent; a "user's home address" is-a "Location":

So if we take the example data flow from earlier then for each of the flows, storages and processes we can construct a set of requirements simply by reading off the corresponding requirements from the structure above.

This leads to an interesting situations where it is possible to construct a set of requirements which are overconstraining. That is simply that we can not build a system that can support everything, for example, one data point might trigger a secret classification and key management, encrypted content etc.

We then need to weaken the requirements such that a system can be constructed according to some economic(!) model. As we weaken or retrench our requirements we are introducing risk into the system,

Aside: Retrenchment - here's a good reference: Banach, Poppleton. Controlling Control Systems: An Application of Evolving Retrenchment. ZB 2002:Formal Specification and Development in Z and BLecture Notes in Computer Science Volume 2272, 2002, pp 42-61.

This gives us our link to risk management and deciding whether each "weakness" we introduce is worth the risk. And as risk can be quantified and we can perform further tasks such as failure mode and effects analysis then we obtain a rigorous method to predict failures and make informed decisions about these. I'll describe more on this in later articles.

Updated Privacy Checklist

So while we're on the subject, here's the checklist I'm currently using as an "aide-mémoire" (I like that term):

It is divided into three parts corresponding to what needs to be established on presentation of the case, what needs to be made during the audit (and this can be repeated as necessary) and what needs to be established to close the audit.

The three "phases" don't necessarily correspond to the underlying process but are more structural to reflect the different phases an audit progresses through. The sign-in only corresponds to the point where a privacy audit team takes over responsibility for the audit; similarly sign-off only corresponds to the point where a privacy audit team wishes to start handing over the results.

Anyway, this is one particular version and the actual implementation is context dependent upon your local management, processes, tools, techniques and system under audit. Modify as required!

Privacy Engineering and Checklists

A colleague brought to my attention a publication from the PbD community on the subject of privacy engineering [1]. Overall the paper I think gives a good introduction into what privacy engineering could be. I say "could by" because PE needs a huge amount of work, from all angles including the deeper mathematical basis as well as basic, software engineering, process, management etc. The current definition given above is:

"...privacy engineering is the discipline of understanding how to include privacy as a non-functional requirement in systems engineering. While privacy may also appear as a functional requirement of a given system (such as the TOR anonymity system), for most systems, privacy is ancillary to the primary purpose
of the system."

Not bad but quite a few of these non-functional requirements become very awkward functional requirements very quickly as the development proceeds.

The part that particular interests me is regarding checklists:

"The checklist approach to privacy protection has been debated. Checklists have become important safety elements in airplane and medical procedures and are quite common in security auditing. However, their utility for privacy remains questionable. It might be possible to design privacy checklists for frequent and standardized use cases, but the breadth of potential projects makes a standard checklist for everything an unlikely tool."

The above cites an earlier article on this blog about moments that changed how particular industries approached safety. There exists possibly a better article on checklists which explains the rationale behind their use.

Partly the paragraph is correct, the utility of checklists in privacy needs to be established, but, given what a checklist is and what is should be designed to achieve is independent of the area. Indeed the next statement that it might be possible to design privacy checklists for frequent and standardised use cases is exactly the cases where checklists do come into their own. Often repeated, critical phases of design - patterns if you like - are the areas where mistakes are frequently made; things are forgotten etc.

There are NO standard checklists - at least not for the expected usages hinted in the paper. If we compare with the WHO surgical checklists which work in similar, very open, high volatility environments there is a capitalised statement on the bottom of the checklist:

THIS CHECKLIST IS NOT INTENDED TO BE COMPREHENSIVE. ADDITIONS AND MODIFICATIONS TO FIT LOCAL PRACTICE ARE ENCOURAGED.

Ignore at your peril.

Indeed the checklist in privacy WILL NOT give you a standardised approach NOR will it give you the tool for checking your system. It WILL give you a list of things that you should remember to complete at particular relevant points. A checklist should be independent of your processes and partially independent of what ever tooling and techniques are being applied to the problem at hand. Furthermore, if the various items on a checklist are not completed it is not an indication that the process can not continue but rather a warning that important facts might not have been established.

For example, on aircraft take-off there is a checklist item for the setting of flaps. This item may not necessarily specify the specific flap settings as this might vary by many conditions and the item can be ignored (and the flaps not set) and take-off can proceed - though this might be a particularly bad idea as in the Spanair Flight 5022 case.

Interestingly in the paper checkilsts seem to be positioned against privacy impact assessments. I could better imagine that in the privacy checklist an entry "PIAs performed" is included, possibly with further qualifications as necessary. It might be considered that such an entry is superfluous - who would forget to do the PIAs? I agree, pilots would never forget to lower flaps prior to take-off or a surgeon forget to check that the he is set up for the correct procedure to be performed....

Maybe the term checklist isn't the best and this is where some confusion arises. At Great Ormond Street hospital the term "Aide-Mémoire" is used to reflect the function of the checklist and avoid the confusion with the "tick-box mentality" and confusion with process.

Privacy is a very wide area without well established and standardised use cases or at least not at the level of detail that say aviation is as is pointed out in the paper we lack. Indeed it is this lack of standardisation that makes the integration of checklists or aide-mémoires much more important.

Some of our experiences with checklists in this area can be found in an earlier posting entitled: Flying Planes, Surgey and Privacy. [2]


References

[1] Shapiro, Cronk, Cavoukian (2014) Privacy Engineering: Proactively Embedding Privacy, by Design.

[2] Ian Oliver, Tomi Kulmala (2013) Flying planes, Surgery and Privacy.

Flying Planes, Surgery and Privacy (external version) from Ian Oliver

Airbus Aircraft Operational Philosophy

I'm reading the Airbus Flight Crew Training Manual for the A320/A321 aircraft at the moment. If anything, to get an understanding of how such things are written and presented. Aircraft are fairly complex systems and as I've already talked about in this blog, many ideas are directly applicable to privacy, security, software engineering etc if we only take time to learn.

Aside: If anyone does own an A320 or equivalent simulator and would like to offer me some time flying (simulated or otherwise) then YES PLEASE!!!.  OK, back to reality now...

A few things struck me as being particularly relevant such as The Operational Golden Rules:

1. The aircraft can be flown like any other aircraft
2. Fly, navigate, communicate - in that order
3. One head up at all times
4. Cross check the accuracy of the FMS
5. Know your FMA at all times 
6. When things don’t go as expected - take over 
7. Use the proper level of automation for the task
8. Practice task sharing and back-up each other

Nothing too surprising there, other than they're stating the obvious.

That's the great thing about the obvious, completely missable...

Note the emphasis on getting the job done without requiring anything other than basic aviation skills; or in our case basic software engineering skills. The emphasis on cross-checking, task sharing and delegation, the use of appropriate techniques and technologies and the note that when things don't go as expected - take over!

As an idea to start off...

1. There is nothing special about this system under development
2. Plan, code, test - in that order
3. Be aware of the wider engineering and requirements context
4. Cross check your system with its original goals
5. ...

The learnings here to our privacy engineering world are deep and profound. Just as an exercise to the reader, look at the procedures being followed for any engineering task and could these be placed into a simple set of operational golden rules as above? There's an interesting comparison waiting to be drawn here...that will have to wait for the moment.

Writing...

Despite writing supposedly being cathartic...ha!...writing anything, from an academic paper to a forthcoming book is not a linear stream of word but a moulding of multiple, parallel, concurrent streams of conciousness into something appearing a whole.

I thought of the analogy with moulding a piece of clay into a work of art...and it works until you realise that piece of clay is probably a piece of s*** that you're holding. At which point you throw it away, wash your hands and feel disgusted and what you've done...

...and then feel more disgusted as to the time you've wasted and the work you've put in and what you've thrown away...warts and all...

It is at that point that you realise that somewhere in that piece of p** was a nugget of gold. So it's on with the rubber gloves to fish out that "clay", find the piece of gold (which may have just been a flash of sunlight if the first place), clean it off and continue with the moulding of your writing until you've achieved something that you're happy to abandon in public view. Just hoping that you've managed to wash off all the smell...

Hemmingway got it right:
The first draft of anything is shit. 

...anyway I'm off to read Pirsig and walk the dog...maybe simultaneously....

Semantics Rant

Worthy of The Daily WTF, and a common occurence when machine types are mixed up with the everyday "types" of things.

Telephone numbers are NOT integers, and money amounts are NOT floating point numbers

Even though the machine types look like these an examination of the ITU-T E.123 document will more than convince that telephone numbers are not integers (or more specifically long unsigned integers). Further proof is that the mathematics operations of addition, subtraction are not defined. Indeed the only operation is equality between two numbers, plus a few other operations to extract different parts, eg: area code, country code etc.

Money amounts similarly: floating point numbers or even decimals have very different properties to the representation and storage of money amounts. In particular the problem of rounding (see also Superman 3 for a humorous take on this). What makes this worse is that support is varying, for example in SQL there are Currency and Money types but these are not supported across all RDBMs, or not provided at all. As Martin Fowler has pointed out, in object-oriented languages this is easily rectified with the provision of a class for money and himself provides the pattern.

Now I feel better...

Privacy Engineering - The Book

Shameless plug, but here's the working cover for my book on privacy engineering aimed at the software engineer whose job it is to construct information systems.

The book will concentrate on the tools and techniques for data flow modelling, information classification and reasoning about information systems from the privacy perspective.Also we will provide details of how to construct a privacy programme, auditing and investigation techniques as well as practical tools such as checklists and a discussion of the pros and cons of various privacy enhancing and enabling technologies.

On being formal, and possibly agile too...

Way back in my past I used to research formal methods for software engineering. Actually I still do, though now most of my time is actually using formal methods to make better software.

Formal methods are nothing more than a collection of languages and techniques for modifying and reasoning about things (models) written in those languages. Some of these techniques encompass how the process of building a system is made. Herein lies one of the first problems encountered by formal methods practitioners and that is the almost constant challenge from some, such as many in the agile community, that seem to be religiously against any form of modelling.

To those who believe that code is the only deliverable and the only thing that matters, well, C++, C, Fortran, Clojure etc are all formal languages, and you're probably using many of the techniques from formal methods right now as your write your code.

Language such as B, Alloy, Z, VDM etc do is provide a method of expressing a model without worrying about certain, awkward details of their implementation or execution.

Indeed what is happening here is that we have languages and techniques that allow you to concentrate on reasoning and thinking about the problem you are trying to solve without getting bogged down in the details of the final implementation.

If your first worry is the implementation language, or the operating system, or which libraries to use, etc, then you're most certainly not solving the problem.

At Nokia we had some very great successes using formal methods in an agile manner for the development of a semantic web infrastructure for the "Internet of Things". Concentrating on what the system had to do and then later worrying about how it was implemented meant that when it did come to the time we needed to architect the components and decide on specific implementation issues we already knew how the system was going to work, what tests we would need to run and what the expected answers were going to be.
Indeed, many of the tests were little more than checking that the code behaved the same as our earlier models - regression testing if you like.

This resulted in a huge decrease in the time spent in coding and the effective removal of nearly all (logic) bugs before even the beta releases. In fact most of the bugs turned out to by typos.

Furthermore, when it came to updating the software with additional features, instead of blindly bolting on a new use case we could reduce most of the new feature requests down to library or convenience functions over the core functionality rather than complicating the design with those "new" features.

This latter point is very important in that even though there is pressure to constantly add new features  (and the Pareto Principle applies here), most new features are really just convenience functions that already exist in the software.

I even remember one system where management demanded so many new features (all specified as their own use cases) that the system actually ended up implementing not only the same feature many times but features to disable the requested feature...

Ultimately formal methods is a discipline of thinking, rather than any technique to develop software. Just as much as Agile is a discipline of development.

To use formal methods does not mean any form of top-down or Waterfall development, it does not mean that one has to use refinement or a language like B or Z or VDM or Alloy etc. Just the simple act of writing a precondition to a function in C, or expressing a simple class diagram in UML, or ER diagram to explain a database schema (SQL or NoSQL) to demonstrate the workings of a system, or to clarify what something mean IS being formal.

The best agile developers I have seen all have formal methods backgrounds. The reasoning is that they already have the discipline and education and the tools to think and reason about their system, even if applied implicitly. Agile depends upon great communication between the developers and the customers and giving those customers exactly what they need in a manner that avoids technical debt (viz. situational awareness).

Whether we like it or not, great software engineering comes from understanding how our craft works at its most fundamental levels - imagine civil engineering without the mathematics of physics (a classic example), or even ballet without an understanding of human movement?

References:

[1] Ian Oliver Experiences of Formal Methods in 'Conventional' Software and Systems Design. FACS 2007 Christmas Workshop: Formal Methods in Industry. BCS London, UK, 17 December 2007 

[2]  Ian Oliver Experiences of Formal Methods in 'Conventional' Software and Systems Design

Formal Methods in Conventional Software Development Environments from Ian Oliver

Privacy Engineers and Privacy Lawyers

Probably on the of the best articles I've seen on the apparent dichotomy between privacy lawyers and the engineers who must build the systems:

Engineers and Lawyers in Privacy Protection: Can We All Just Get Along?

By Peter Swire,and Annie Antón January 13, 2014

Actually I'd add a 3rd group called privacy advocates who tend to side with the lawyers and believe that the engineers are there to do their bidding.

But let's take a specific example as mentioned in the text, that of data minimisation. Actually it turns out that [software] engineers rarely gather too much - it really isn't in our nature to overcomplicate the already complicated process of building information systems. Indeed one of the ways to upset engineers is to repeatedly tell them not to collect data they aren't collecting in the first place. Each new data point usually involves additional validation, verification, tests and obviously more code. More code => more bugs => more test => more time => more expense etc...

But we also see other problems such as the emergence of the privacy cabal - a group of predominantly lawyers and advocates who do not understand the discipline and complexities  of engineering. Again the article above quite well explains this. Indeed Jim Adler in his PII2012 talk called this the Privacy-Industrial Complex: a mechanism for churning our policies, guidelines, edicts on the topic of privacy but without any basis in engineering reality. The engineers role is reduced to a mere bystander, a group of people to be handed orders from the cabal and privacy priesthood on high.

When things go wrong, it is invariably the engineers' fault, while the priests of privacy claim that their policies conform to best practice and take solice in the Privacy by Design Commandments. Take Target for example, their POS system collected only necessary data - credit card numbers  - was this an example of applying Fair Information Policy Principles? Is collecting and processing credit card numbers necessary for processing credit card numbers? Do our privacy principles accurately capture subtle requirements such as caching data in memory, types of encryption algorithm required, the human-computer interface etc Aside: OK, Target failed in many respects and the results of that investigation are to be seen.

Requirements such as don't collect PII ... is an IP address PII? How do I stop collecting this when the very protocols of the internet require such addresses just as the postal system relies upon physical addresses.

The position of the emerging privacy engineer will become one of the most important positions in the field of privacy. A group of people who understand the fundamentals of information systems from their engineering to their mathematical foundataions is critically required. Ignore these foundations and privacy becomes an hand-waving, powerpoint generating inconvenience to be humored and tolerated rather than an integral part of the business ecosystem.

When lawyers and engineers work together AS EQUALS we get some truly AMAZING work done. It happens rarely and it takes a lot of work from both sides just to build a common framework of understanding. Engineers have the ability to properly deconstruct and understand the finer workings of privacy compliance - ignore this as privacy will remain as we described above: a tolerated inconvenience.

Getting Started with Clojure

I'm starting to enjoy using Clojure - the Lisp built on top of Java if you like. Great set of libraries, available and relevant documentation, a good base of users and plenty of examples.

I admit I'm already familiar with the functional programming style and have been for many years since discovering ML, Hope and Gofer many, many years ago, so I guess moving back to the one true language - Lisp - isn't so difficult, and anyway I don't want to talk about functional programming.

As everything seems to be about web based services, it tends to always be an quick experiment for me to build a simple web based service to get to grips with a language and its support. I must admit I loved Opa, but was let down by the support; Ruby on Rails I never really liked and let's not discuss Java and all its frameworks. Anyway, Clojure has been getting a lot of attention and a couple of colleagues of mine have had some very favourable things to say about it (one is a Lisp-fanatic however...) So here goes.

First go get Leiningen, the rather cool Clojure project automator. Dump that in a place where everyone can see it. I'm using Debian 7, so it goes in /usr/local/bin for me. I assume you're already familiar with things like this. Create a new Clojure project, which creates a whole bunch of directories under ./test

ian@deb1:~$ lein new test
Generating a project called test based on the 'default' template.
To see other templates (app, lein plugin, etc), try `lein help new`.
ian@ian@deb1:~$ ls -l test
total 36
drwxr-xr-x 2 ian ian  4096 Jan 15 10:52 doc
-rw-r--r-- 1 ian ian 11220 Jan 15 10:52 LICENSE
-rw-r--r-- 1 ian ian   264 Jan 15 10:52 project.clj
-rw-r--r-- 1 ian ian   230 Jan 15 10:52 README.md
drwxr-xr-x 2 ian ian  4096 Jan 15 10:52 resources
drwxr-xr-x 3 ian ian  4096 Jan 15 10:52 src
drwxr-xr-x 3 ian ian  4096 Jan 15 10:52 test

So what we're going to do is write a small service that queries a database from a call over HTTP or via a browser.

First however, we need to make sure we've a database somewhere. I'm using MariaDB, you could use MySQL or any other RBMS system, but you'll need to check some specifics on how to connect. I've created a database called testdb, a single table and populated it with some data. You'll need to use whatever tools your database provides to do this, but for MariaDB (and MySQL I guess) it is the following:

ian@ian@deb1:~/dbtest$ mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 28
Server version: 5.5.34-MariaDB-1~wheezy-log mariadb.org binary distribution
Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> create database fred;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> use "fred";
Database changed
MariaDB [fred]> create table ids (id integer, name varchar(50) );
Query OK, 0 rows affected (0.17 sec)
MariaDB [fred]> insert into ids values (1,"Alice");
Query OK, 1 row affected (0.02 sec)
MariaDB [fred]> insert into ids values (2,"Bob");
Query OK, 1 row affected (0.01 sec)
MariaDB [fred]> insert into ids values (3,"Eve");
Query OK, 1 row affected (0.01 sec)
MariaDB [fred]> commit;
Query OK, 0 rows affected (0.00 sec)

Aside: yes, I should be shot for using the database root user and not bothering about security...agreed! Don't do it and go read up on such things.

Now the two important files are project.clj and core.clj ... use your favourite text editor to open these:

ian@ian@deb1:~$ cd dbtest
ian@ian@deb1:~/dbtest$ gedit project.clj src/dbtest/core.clj &

Edit the file project.clj so it looks like this:

(defproject dbtest "v0.1 DBTest"
  :description "Simple Test Application"
  :url "http://http://ijosblog.blogspot.fi/"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [org.clojure/java.jdbc "0.2.3"]
                 [compojure "1.1.6"]
                 [mysql/mysql-connector-java "5.1.18"]]
  :plugins [[lein-ring "0.8.10"]]
  :ring {:handler dbtest.core/app }
  :profiles {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
                        [ring-mock "0.1.5"]]}}
)

Don't worry about the description, url and license properties or the name of the project. The important things here are the dependencies, plugins, profiles and ring.

These describe the packages that we'll be including in our project, think of them as #includes in C/C++. What we're stating here is that we require Clojure version 1.5.1, Java JDBC , the MySQL connector (works with MariaDB) and a package called Compojure which will handle requests. Additionally we have a plugin ca lled lein-ring which handles the low-level details of HTTP calls plus provides a server to receive the calls. The line stating with :ring states the function that will be called on the start up of the application. The specifics aren't too important at the moment, we can learn about that later.

The actual code that we will run is held in the core.clj file by default. There are clues to Clojure's structuring and packaging mechanisms in the above code and in the directory structure. Firstly the we define the namespace and the packages we wish to include:

(ns dbtest.core  (:use [compojure.core]        [clojure.test] )      (:require [compojure.handler :as handler]            [compojure.route :as route] ))(use 'clojure.java.jdbc)

Then we define the function helloworld:

(defn helloworld []
  "Hello World")

and the functions that handle requests from the client

(defroutes app-routes
  (GET "/" [] ( helloworld ))
  (route/resources "/")
  (route/not-found "Not Found"))

(def app
  (handler/site app-routes))

And that is effectively everything we need to do to get something running. But before you do run, it is necessary to make sure we have all the libraries installed and lein does this for us by consulting the project.clj file, so run:

ian@ian@deb1:~/dbtest$ lein deps

and let it download everything you need. You generally only have to do ths when you add libraries to the system. Once that has completed (there'll be lots of messages about downloads, or nothing if you've already got everything) start the server:

ian@ian@deb1:~/dbtest$ lein ring server
2014-01-15 11:12:17.221:INFO:oejs.Server:jetty-7.6.8.v20121106
2014-01-15 11:12:17.345:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:3000
Started server on port 3000

A browser window should pop up, but if not start your favourity browser, eg: firefox and point it at http://127.0.0.1:3000 and you should get a page starting Hello World. NB: 127.0.0.1 always resolves to the local machine, so if you have the browser running on a seperate machine (virtual or not) then make sure you use the correct IP address. For the brave you can always use curl to get the body and headers by running the following in a seperate terminal window:

ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000
Hello World
ian@ian@deb1:~/dbtest$ curl -I  http://127.0.0.1:3000
HTTP/1.1 200 OK
Date: Wed, 15 Jan 2014 09:19:09 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 0
Server: Jetty(7.6.8.v20121106)

After you're happy, kill the server with ctrl-c.

Let's now connect to the database by adding the following code between the (ns...) and (use...) and our hello world function:

(let [db-host "localhost"
      db-port 3306
      db-name "testdb"]

  (def db {:classname "com.mysql.jdbc.Driver" ; must be in classpath
           :subprotocol "mysql"
           :subname (str "//" db-host ":" db-port "/" db-name)
           ; Any additional keys are passed to the driver
           ; as driver-specific properties.
           :user "root"
           :password "badpassword"}))

Firstly localhost is the local machine again. I'm running everything on a single machine, but if the database were elsewhere then obviously I'd need to the use the IP address of that machine. And if I were being clever then I'd pull all this in from some configuration file...again, later...

The user and password are used to log in to that specific database (testdb), which if you remember I created using the MariaDB root user (bad!) and also note the horrificly insecure password I was using. Oh and it is an even worse idea to hard code it in plain text in your source file. Don't say you weren't warned. DO NOT DO THIS IN ANYTHNG OTHER THAN A TRIVIAL EXAMPLE, and even then it is probably a bad idea.

After the above code which connects to our database, we add a simple function:

(defn list-all []  (with-connection db    (with-query-results rs ["select * from ids"]      (doall rs))))

This states to use the connection details stored in the db variable and make a query on the database and store the results in as a list called rs and then process it. In this case we're forcing the list to be evaluated using the doall function over the list rs. Clojure is a lazy functional programming language like Haskell, or at least has lazy features.

Now modify the app-routes function so:

(defroutes app-routes
  (GET "/" [] ( helloworld ))
  (GET "/listall" [] ( list-all ))
  (route/resources "/")
  (route/not-found "Not Found"))

Don't forget to restart the server:

ian@ian@deb1:~/dbtest$ lein ring server
2014-01-15 11:12:17.221:INFO:oejs.Server:jetty-7.6.8.v20121106
2014-01-15 11:12:17.345:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:3000
Started server on port 3000

and in a seperate terminal (or put the URL below into your browser)

ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/listall
{:name "Alice", :id 1}{:name "Bob", :id 2}{:name "Eve", :id 3}

And there you have Clojure returning the contents of the database. After this, kill the server again (note: there is a way to get lein to automatically rebuild and restart the server...exercise left to reader ;-)

Now add the function:

(defn get-by-id [the_id]
  (with-connection db
   (with-query-results rs ["select * from ids where id=?" the_id ]
     (doall rs))))

Aside: the above probably could do with some validation to protect against SQL injection and other bad data (cf: obxkcd)

and modify app-routes so it contains an additional line:

  (GET "/id/:the_id" [the_id] ( get-by-id the_id )) 

and restart the server as before and test with curl (or your browser):

ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/id/1
{:name "Alice", :id 1}
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/id/2
{:name "Bob", :id 2}
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/id/3
{:name "Eve", :id 3}
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/id/4
Not Found

And that's it, a "fully" functioning web service of sorts. 0/10 probably for style and -100000s for security but quite cool for so few lines of code.

--

Here's the final code for the core.clj file:

(ns dbtest.core  (:use [compojure.core]        [clojure.test])      (:require [compojure.handler :as handler]            [compojure.route :as route]          ))
(use 'clojure.java.jdbc) (let [db-host "localhost"    ;; change this as necessary      db-port 3306           ;; change this as necessary      db-name "testdb"]      ;; change this as necessary   (def db {:classname "com.mysql.jdbc.Driver" ; must be in classpath           :subprotocol "mysql"           :subname (str "//" db-host ":" db-port "/" db-name)           ; Any additional keys are passed to the driver           ; as driver-specific properties.           :user "root"           :password "badpassword"}))     ;; change this as necessary!

(defn list-all []  (with-connection db    (with-query-results rs ["select * from ids"]      (doall rs))))

(defn get-by-id [the_id]  (with-connection db    (with-query-results rs ["select * from ids where id=?" the_id ]      (doall rs))))
(defn helloworld []  "Hello World")
(defroutes app-routes  (GET "/" [] ( helloworld ))  (GET "/listall" [] ( list-all ))  (GET "/id/:the_id" [the_id] ( get-by-id the_id ))   (route/resources "/")  (route/not-found "Not Found"))
(def app  (handler/site app-routes))

Child Protection Software

Something of a hiatus, been busy with the book, formalising a major company's privacy policies and a side order of Clojure programming (see next post). All-in-all, busy. However I just want to turn to some experiences with software for child proofing a computer, or a tablet if you prefer. Protecting children from all the nasties of the Internet seems to be quite a moral crusade by some. Massive investments in filtering and censorship software and schemes seem to take precedence over good old-fashioned education - you know, that thing parents are supposed to do?

I've been working through some of these systems for child protection, ostensibly running on Android. Generally they all suck badly making the device almost impossible to use and administer and causing more headaches for the parents...just a general observation from asking many on the subject, including a few child protection experts.

One thing particularly struck me about some of the software for this kind of protection, they all seem to be gathering an awful lot of information about the child, either through reporting back statistics (including location) or by acting as proxies to content. In the latter case you are also at the mercy of the protection software provider with regards to what they're filtering.

One system asked me to register with a huge amount of personal details: my details, partner details, email addresses, the childrens' details, addresses, dates of birth, gender, preferences, interests, hobbies etc (marketers' dream), and then informed my through an extraordinarily lengthy terms and condition and privacy policy that they would collect the location of the tablet/phone/computer each time the software was activated, shut down, used. Finally they topped this off with a statement about how some data would be `anonymously' used by (unnamed) 3rd party companies for providing a better service.

Suffice to say I didn't install it and reverted to supervision, education and being a parent.

Earth and Moon from Juno

Somewhat stunning (understatment!) video made from Juno's low resolution camera during its Earth fly-by back in October:


Thanks to Phil Plait of Bad Astronomy once again...and just to think that every human that has ever lived existed in and no human has even travelled beyond the frames in that video...

An Academic Survey

I received this morning an email asking to me participate in a survey about "perceptions scientists have of information providers in scientific, technical and medical research fields.."

I'm always a little suspicious of such things but this does seem genuine-ish, if a little badly written; actually I was wondering if they were going to ask me to be a proxy to transfer 50 million USD from an account in the Central African Republic to somewhere on behalf of the Emporer of Botswana?

Do they really have to be so "secretive" in the wording too? I wonder what paper they are referring to, I published quite a few in 2010? What was the "careful selection criteria?"

Here's the mail:

Dear Dr. Oliver ,

We are contacting you because you are the corresponding author on a paper that was published in 2010. We are conducting a survey about the perceptions scientists have of information providers in scientific, technical and medical research fields. As only a carefully selected sample of scientists and practitioners have been chosen for the study, your feedback is very valuable to us.

The survey is being conducted by a scientific, technical and medical publisher who will be revealed at the end of the survey. Under the terms of the Market Research Society Code of Conduct, it will not lead to any sales follow up and no individual (or organization) will be identified. Your results will be kept confidential and used only for research purposes.

We would be grateful if you could spare approximately 15 minutes to complete this survey. The closing date for the survey is 23rd December 2013.

To start the survey, please click ONCE on the link below (or paste it into your browser):

[link redacted]

Please do not reply to this e-mail as the inbox is not monitored. If you are having trouble with this survey you can let us know [redacted]) and we will address any technical problems as quickly as we can.

Thank you very much for your time, we really value your input.

It mentions the "Market Research Society Code of Conduct" and ensures me that my data will only be used for the primary purposes of the survey; no marketing etc...The company sending this is Confirmit, a Norwegian based company that conducts market research - just wonder why they're quoting the MRSC code of conduct presumably given that the MRSC is a UK organisation as far as I can tell.



I'd actually like to know what their privacy policy is, and how they interact with their customers...for example, do they anonymise by hashing unique identifiers? Wwhat is the full set of data that does flow to the customer? What to Confirmit keep and for how long?

Anyway, I contacted MRSC to find out whether this is genuine or not and who is actually using this data.

A few other searches suggests that this might be a request by Elsevier to find out why authors are so upset with them. I guess it isn't a secret that I frequent the category theory lists and there might be one or two there who might not be friends with Elsevier....

So, went to Elsevier's web pages to find some contact details to find out whether they were asking and again whether this is something genuine or some evil spam...clicking on "Help & Contact" gives, well....

I'll let you know how I get on if I get any replies....