Way back in my past I used to research formal methods for software engineering. Actually I still do, though now most of my time is actually using formal methods to make better software.
Formal methods are nothing more than a collection of languages and techniques for modifying and reasoning about things (models) written in those languages. Some of these techniques encompass how the process of building a system is made. Herein lies one of the first problems encountered by formal methods practitioners and that is the almost constant challenge from some, such as many in the agile community, that seem to be religiously against any form of modelling.
To those who believe that code is the only deliverable and the only thing that matters, well, C++, C, Fortran, Clojure etc are all formal languages, and you're probably using many of the techniques from formal methods right now as your write your code.
Language such as B, Alloy, Z, VDM etc do is provide a method of expressing a model without worrying about certain, awkward details of their implementation or execution.
Indeed what is happening here is that we have languages and techniques that allow you to concentrate on reasoning and thinking about the problem you are trying to solve without getting bogged down in the details of the final implementation.
If your first worry is the implementation language, or the operating system, or which libraries to use, etc, then you're most certainly not solving the problem.
At Nokia we had some very great successes using formal methods in an agile manner for the development of a semantic web infrastructure for the "Internet of Things". Concentrating on what the system had to do and then later worrying about how it was implemented meant that when it did come to the time we needed to architect the components and decide on specific implementation issues we already knew how the system was going to work, what tests we would need to run and what the expected answers were going to be.
Indeed, many of the tests were little more than checking that the code behaved the same as our earlier models - regression testing if you like.
This resulted in a huge decrease in the time spent in coding and the effective removal of nearly all (logic) bugs before even the beta releases. In fact most of the bugs turned out to by typos.
Furthermore, when it came to updating the software with additional features, instead of blindly bolting on a new use case we could reduce most of the new feature requests down to library or convenience functions over the core functionality rather than complicating the design with those "new" features.
This latter point is very important in that even though there is pressure to constantly add new features (and the Pareto Principle applies here), most new features are really just convenience functions that already exist in the software.
I even remember one system where management demanded so many new features (all specified as their own use cases) that the system actually ended up implementing not only the same feature many times but features to disable the requested feature...
Ultimately formal methods is a discipline of thinking, rather than any technique to develop software. Just as much as Agile is a discipline of development.
To use formal methods does not mean any form of top-down or Waterfall development, it does not mean that one has to use refinement or a language like B or Z or VDM or Alloy etc. Just the simple act of writing a precondition to a function in C, or expressing a simple class diagram in UML, or ER diagram to explain a database schema (SQL or NoSQL) to demonstrate the workings of a system, or to clarify what something mean IS being formal.
The best agile developers I have seen all have formal methods backgrounds. The reasoning is that they already have the discipline and education and the tools to think and reason about their system, even if applied implicitly. Agile depends upon great communication between the developers and the customers and giving those customers exactly what they need in a manner that avoids technical debt (viz. situational awareness).
Whether we like it or not, great software engineering comes from understanding how our craft works at its most fundamental levels - imagine civil engineering without the mathematics of physics (a classic example), or even ballet without an understanding of human movement?
References:
[1] Ian Oliver Experiences of Formal Methods in 'Conventional' Software and Systems Design. FACS 2007 Christmas Workshop: Formal Methods in Industry. BCS London, UK, 17 December 2007
[2] Ian Oliver Experiences of Formal Methods in 'Conventional' Software and Systems Design
A collection of discussions, links, stories, news and whatever else I find interesting in the fields of computing, information, science, privacy, semantics, mathematics and so on...
Thursday, 23 January 2014
Tuesday, 21 January 2014
Privacy Engineers and Privacy Lawyers
Probably on the of the best articles I've seen on the apparent dichotomy between privacy lawyers and the engineers who must build the systems:
Engineers and Lawyers in Privacy Protection: Can We All Just Get Along?
Engineers and Lawyers in Privacy Protection: Can We All Just Get Along?
By Peter Swire,and Annie Antón January 13, 2014
Actually I'd add a 3rd group called privacy advocates who tend to side with the lawyers and believe that the engineers are there to do their bidding.
Actually I'd add a 3rd group called privacy advocates who tend to side with the lawyers and believe that the engineers are there to do their bidding.
But let's take a specific example as mentioned in the text, that of data minimisation. Actually it turns out that [software] engineers rarely gather too much - it really isn't in our nature to overcomplicate the already complicated process of building information systems. Indeed one of the ways to upset engineers is to repeatedly tell them not to collect data they aren't collecting in the first place. Each new data point usually involves additional validation, verification, tests and obviously more code. More code => more bugs => more test => more time => more expense etc...
But we also see other problems such as the emergence of the privacy cabal - a group of predominantly lawyers and advocates who do not understand the discipline and complexities of engineering. Again the article above quite well explains this. Indeed Jim Adler in his PII2012 talk called this the Privacy-Industrial Complex: a mechanism for churning our policies, guidelines, edicts on the topic of privacy but without any basis in engineering reality. The engineers role is reduced to a mere bystander, a group of people to be handed orders from the cabal and privacy priesthood on high.
When things go wrong, it is invariably the engineers' fault, while the priests of privacy claim that their policies conform to best practice and take solice in the Privacy by Design Commandments. Take Target for example, their POS system collected only necessary data - credit card numbers - was this an example of applying Fair Information Policy Principles? Is collecting and processing credit card numbers necessary for processing credit card numbers? Do our privacy principles accurately capture subtle requirements such as caching data in memory, types of encryption algorithm required, the human-computer interface etc Aside: OK, Target failed in many respects and the results of that investigation are to be seen.
Requirements such as don't collect PII ... is an IP address PII? How do I stop collecting this when the very protocols of the internet require such addresses just as the postal system relies upon physical addresses.
The position of the emerging privacy engineer will become one of the most important positions in the field of privacy. A group of people who understand the fundamentals of information systems from their engineering to their mathematical foundataions is critically required. Ignore these foundations and privacy becomes an hand-waving, powerpoint generating inconvenience to be humored and tolerated rather than an integral part of the business ecosystem.
When lawyers and engineers work together AS EQUALS we get some truly AMAZING work done. It happens rarely and it takes a lot of work from both sides just to build a common framework of understanding. Engineers have the ability to properly deconstruct and understand the finer workings of privacy compliance - ignore this as privacy will remain as we described above: a tolerated inconvenience.
Wednesday, 15 January 2014
Getting Started with Clojure
I'm starting to enjoy using Clojure - the Lisp built on top of Java if you like. Great set of libraries, available and relevant documentation, a good base of users and plenty of examples.
I admit I'm already familiar with the functional programming style and have been for many years since discovering ML, Hope and Gofer many, many years ago, so I guess moving back to the one true language - Lisp - isn't so difficult, and anyway I don't want to talk about functional programming.
As everything seems to be about web based services, it tends to always be an quick experiment for me to build a simple web based service to get to grips with a language and its support. I must admit I loved Opa, but was let down by the support; Ruby on Rails I never really liked and let's not discuss Java and all its frameworks. Anyway, Clojure has been getting a lot of attention and a couple of colleagues of mine have had some very favourable things to say about it (one is a Lisp-fanatic however...) So here goes.
First go get Leiningen, the rather cool Clojure project automator. Dump that in a place where everyone can see it. I'm using Debian 7, so it goes in /usr/local/bin for me. I assume you're already familiar with things like this. Create a new Clojure project, which creates a whole bunch of directories under ./test
First however, we need to make sure we've a database somewhere. I'm using MariaDB, you could use MySQL or any other RBMS system, but you'll need to check some specifics on how to connect. I've created a database called testdb, a single table and populated it with some data. You'll need to use whatever tools your database provides to do this, but for MariaDB (and MySQL I guess) it is the following:
Now the two important files are project.clj and core.clj ... use your favourite text editor to open these:
These describe the packages that we'll be including in our project, think of them as #includes in C/C++. What we're stating here is that we require Clojure version 1.5.1, Java JDBC , the MySQL connector (works with MariaDB) and a package called Compojure which will handle requests. Additionally we have a plugin ca lled lein-ring which handles the low-level details of HTTP calls plus provides a server to receive the calls. The line stating with :ring states the function that will be called on the start up of the application. The specifics aren't too important at the moment, we can learn about that later.
The actual code that we will run is held in the core.clj file by default. There are clues to Clojure's structuring and packaging mechanisms in the above code and in the directory structure. Firstly the we define the namespace and the packages we wish to include:
Let's now connect to the database by adding the following code between the (ns...) and (use...) and our hello world function:
The user and password are used to log in to that specific database (testdb), which if you remember I created using the MariaDB root user (bad!) and also note the horrificly insecure password I was using. Oh and it is an even worse idea to hard code it in plain text in your source file. Don't say you weren't warned. DO NOT DO THIS IN ANYTHNG OTHER THAN A TRIVIAL EXAMPLE, and even then it is probably a bad idea.
After the above code which connects to our database, we add a simple function:
Now modify the app-routes function so:
Now add the function:
and modify app-routes so it contains an additional line:
--
I admit I'm already familiar with the functional programming style and have been for many years since discovering ML, Hope and Gofer many, many years ago, so I guess moving back to the one true language - Lisp - isn't so difficult, and anyway I don't want to talk about functional programming.
As everything seems to be about web based services, it tends to always be an quick experiment for me to build a simple web based service to get to grips with a language and its support. I must admit I loved Opa, but was let down by the support; Ruby on Rails I never really liked and let's not discuss Java and all its frameworks. Anyway, Clojure has been getting a lot of attention and a couple of colleagues of mine have had some very favourable things to say about it (one is a Lisp-fanatic however...) So here goes.
First go get Leiningen, the rather cool Clojure project automator. Dump that in a place where everyone can see it. I'm using Debian 7, so it goes in /usr/local/bin for me. I assume you're already familiar with things like this. Create a new Clojure project, which creates a whole bunch of directories under ./test
ian@deb1:~$ lein new testSo what we're going to do is write a small service that queries a database from a call over HTTP or via a browser.
Generating a project called test based on the 'default' template.
To see other templates (app, lein plugin, etc), try `lein help new`.
ian@ian@deb1:~$ ls -l test
total 36
drwxr-xr-x 2 ian ian 4096 Jan 15 10:52 doc
-rw-r--r-- 1 ian ian 11220 Jan 15 10:52 LICENSE
-rw-r--r-- 1 ian ian 264 Jan 15 10:52 project.clj
-rw-r--r-- 1 ian ian 230 Jan 15 10:52 README.md
drwxr-xr-x 2 ian ian 4096 Jan 15 10:52 resources
drwxr-xr-x 3 ian ian 4096 Jan 15 10:52 src
drwxr-xr-x 3 ian ian 4096 Jan 15 10:52 test
First however, we need to make sure we've a database somewhere. I'm using MariaDB, you could use MySQL or any other RBMS system, but you'll need to check some specifics on how to connect. I've created a database called testdb, a single table and populated it with some data. You'll need to use whatever tools your database provides to do this, but for MariaDB (and MySQL I guess) it is the following:
ian@ian@deb1:~/dbtest$ mysql -u root -pAside: yes, I should be shot for using the database root user and not bothering about security...agreed! Don't do it and go read up on such things.
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 28
Server version: 5.5.34-MariaDB-1~wheezy-log mariadb.org binary distribution
Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> create database fred;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> use "fred";
Database changed
MariaDB [fred]> create table ids (id integer, name varchar(50) );
Query OK, 0 rows affected (0.17 sec)
MariaDB [fred]> insert into ids values (1,"Alice");
Query OK, 1 row affected (0.02 sec)
MariaDB [fred]> insert into ids values (2,"Bob");
Query OK, 1 row affected (0.01 sec)
MariaDB [fred]> insert into ids values (3,"Eve");
Query OK, 1 row affected (0.01 sec)
MariaDB [fred]> commit;
Query OK, 0 rows affected (0.00 sec)
Now the two important files are project.clj and core.clj ... use your favourite text editor to open these:
ian@ian@deb1:~$ cd dbtestEdit the file project.clj so it looks like this:
ian@ian@deb1:~/dbtest$ gedit project.clj src/dbtest/core.clj &
(defproject dbtest "v0.1 DBTest"Don't worry about the description, url and license properties or the name of the project. The important things here are the dependencies, plugins, profiles and ring.
:description "Simple Test Application"
:url "http://http://ijosblog.blogspot.fi/"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
[org.clojure/java.jdbc "0.2.3"]
[compojure "1.1.6"]
[mysql/mysql-connector-java "5.1.18"]]
:plugins [[lein-ring "0.8.10"]]
:ring {:handler dbtest.core/app }
:profiles {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
[ring-mock "0.1.5"]]}}
)
These describe the packages that we'll be including in our project, think of them as #includes in C/C++. What we're stating here is that we require Clojure version 1.5.1, Java JDBC , the MySQL connector (works with MariaDB) and a package called Compojure which will handle requests. Additionally we have a plugin ca lled lein-ring which handles the low-level details of HTTP calls plus provides a server to receive the calls. The line stating with :ring states the function that will be called on the start up of the application. The specifics aren't too important at the moment, we can learn about that later.
The actual code that we will run is held in the core.clj file by default. There are clues to Clojure's structuring and packaging mechanisms in the above code and in the directory structure. Firstly the we define the namespace and the packages we wish to include:
(ns dbtest.core (:use [compojure.core] [clojure.test] ) (:require [compojure.handler :as handler] [compojure.route :as route] ))(use 'clojure.java.jdbc)Then we define the function helloworld:
(defn helloworld []and the functions that handle requests from the client
"Hello World")
(defroutes app-routes
(GET "/" [] ( helloworld ))
(route/resources "/")
(route/not-found "Not Found"))
(def appAnd that is effectively everything we need to do to get something running. But before you do run, it is necessary to make sure we have all the libraries installed and lein does this for us by consulting the project.clj file, so run:
(handler/site app-routes))
ian@ian@deb1:~/dbtest$ lein depsand let it download everything you need. You generally only have to do ths when you add libraries to the system. Once that has completed (there'll be lots of messages about downloads, or nothing if you've already got everything) start the server:
ian@ian@deb1:~/dbtest$ lein ring serverA browser window should pop up, but if not start your favourity browser, eg: firefox and point it at http://127.0.0.1:3000 and you should get a page starting Hello World. NB: 127.0.0.1 always resolves to the local machine, so if you have the browser running on a seperate machine (virtual or not) then make sure you use the correct IP address. For the brave you can always use curl to get the body and headers by running the following in a seperate terminal window:
2014-01-15 11:12:17.221:INFO:oejs.Server:jetty-7.6.8.v20121106
2014-01-15 11:12:17.345:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:3000
Started server on port 3000
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000After you're happy, kill the server with ctrl-c.
Hello World
ian@ian@deb1:~/dbtest$ curl -I http://127.0.0.1:3000
HTTP/1.1 200 OK
Date: Wed, 15 Jan 2014 09:19:09 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 0
Server: Jetty(7.6.8.v20121106)
Let's now connect to the database by adding the following code between the (ns...) and (use...) and our hello world function:
(let [db-host "localhost"Firstly localhost is the local machine again. I'm running everything on a single machine, but if the database were elsewhere then obviously I'd need to the use the IP address of that machine. And if I were being clever then I'd pull all this in from some configuration file...again, later...
db-port 3306
db-name "testdb"]
(def db {:classname "com.mysql.jdbc.Driver" ; must be in classpath
:subprotocol "mysql"
:subname (str "//" db-host ":" db-port "/" db-name)
; Any additional keys are passed to the driver
; as driver-specific properties.
:user "root"
:password "badpassword"}))
The user and password are used to log in to that specific database (testdb), which if you remember I created using the MariaDB root user (bad!) and also note the horrificly insecure password I was using. Oh and it is an even worse idea to hard code it in plain text in your source file. Don't say you weren't warned. DO NOT DO THIS IN ANYTHNG OTHER THAN A TRIVIAL EXAMPLE, and even then it is probably a bad idea.
After the above code which connects to our database, we add a simple function:
(defn list-all [] (with-connection db (with-query-results rs ["select * from ids"] (doall rs))))This states to use the connection details stored in the db variable and make a query on the database and store the results in as a list called rs and then process it. In this case we're forcing the list to be evaluated using the doall function over the list rs. Clojure is a lazy functional programming language like Haskell, or at least has lazy features.
Now modify the app-routes function so:
(defroutes app-routesDon't forget to restart the server:
(GET "/" [] ( helloworld ))
(GET "/listall" [] ( list-all ))
(route/resources "/")
(route/not-found "Not Found"))
ian@ian@deb1:~/dbtest$ lein ring serverand in a seperate terminal (or put the URL below into your browser)
2014-01-15 11:12:17.221:INFO:oejs.Server:jetty-7.6.8.v20121106
2014-01-15 11:12:17.345:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:3000
Started server on port 3000
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/listallAnd there you have Clojure returning the contents of the database. After this, kill the server again (note: there is a way to get lein to automatically rebuild and restart the server...exercise left to reader ;-)
{:name "Alice", :id 1}{:name "Bob", :id 2}{:name "Eve", :id 3}
Now add the function:
(defn get-by-id [the_id]Aside: the above probably could do with some validation to protect against SQL injection and other bad data (cf: obxkcd)
(with-connection db
(with-query-results rs ["select * from ids where id=?" the_id ]
(doall rs))))
and modify app-routes so it contains an additional line:
(GET "/id/:the_id" [the_id] ( get-by-id the_id ))and restart the server as before and test with curl (or your browser):
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/id/1And that's it, a "fully" functioning web service of sorts. 0/10 probably for style and -100000s for security but quite cool for so few lines of code.
{:name "Alice", :id 1}
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/id/2
{:name "Bob", :id 2}
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/id/3
{:name "Eve", :id 3}
ian@ian@deb1:~/dbtest$ curl http://127.0.0.1:3000/id/4
Not Found
--
Here's the final code for the core.clj file:
(ns dbtest.core (:use [compojure.core] [clojure.test]) (:require [compojure.handler :as handler] [compojure.route :as route] ))
(use 'clojure.java.jdbc) (let [db-host "localhost" ;; change this as necessary db-port 3306 ;; change this as necessary db-name "testdb"] ;; change this as necessary (def db {:classname "com.mysql.jdbc.Driver" ; must be in classpath :subprotocol "mysql" :subname (str "//" db-host ":" db-port "/" db-name) ; Any additional keys are passed to the driver ; as driver-specific properties. :user "root" :password "badpassword"})) ;; change this as necessary!
(defn list-all [] (with-connection db (with-query-results rs ["select * from ids"] (doall rs))))
(defn get-by-id [the_id] (with-connection db (with-query-results rs ["select * from ids where id=?" the_id ] (doall rs))))
(defn helloworld [] "Hello World")
(defroutes app-routes (GET "/" [] ( helloworld )) (GET "/listall" [] ( list-all )) (GET "/id/:the_id" [the_id] ( get-by-id the_id )) (route/resources "/") (route/not-found "Not Found"))
(def app (handler/site app-routes))
Child Protection Software
Something of a hiatus, been busy with the book, formalising a major company's privacy policies and a side order of Clojure programming (see next post). All-in-all, busy. However I just want to turn to some experiences with software for child proofing a computer, or a tablet if you prefer. Protecting children from all the nasties of the Internet seems to be quite a moral crusade by some. Massive investments in filtering and censorship software and schemes seem to take precedence over good old-fashioned education - you know, that thing parents are supposed to do?
I've been working through some of these systems for child protection, ostensibly running on Android. Generally they all suck badly making the device almost impossible to use and administer and causing more headaches for the parents...just a general observation from asking many on the subject, including a few child protection experts.
One thing particularly struck me about some of the software for this kind of protection, they all seem to be gathering an awful lot of information about the child, either through reporting back statistics (including location) or by acting as proxies to content. In the latter case you are also at the mercy of the protection software provider with regards to what they're filtering.
One system asked me to register with a huge amount of personal details: my details, partner details, email addresses, the childrens' details, addresses, dates of birth, gender, preferences, interests, hobbies etc (marketers' dream), and then informed my through an extraordinarily lengthy terms and condition and privacy policy that they would collect the location of the tablet/phone/computer each time the software was activated, shut down, used. Finally they topped this off with a statement about how some data would be `anonymously' used by (unnamed) 3rd party companies for providing a better service.
Suffice to say I didn't install it and reverted to supervision, education and being a parent.
I've been working through some of these systems for child protection, ostensibly running on Android. Generally they all suck badly making the device almost impossible to use and administer and causing more headaches for the parents...just a general observation from asking many on the subject, including a few child protection experts.
One thing particularly struck me about some of the software for this kind of protection, they all seem to be gathering an awful lot of information about the child, either through reporting back statistics (including location) or by acting as proxies to content. In the latter case you are also at the mercy of the protection software provider with regards to what they're filtering.
One system asked me to register with a huge amount of personal details: my details, partner details, email addresses, the childrens' details, addresses, dates of birth, gender, preferences, interests, hobbies etc (marketers' dream), and then informed my through an extraordinarily lengthy terms and condition and privacy policy that they would collect the location of the tablet/phone/computer each time the software was activated, shut down, used. Finally they topped this off with a statement about how some data would be `anonymously' used by (unnamed) 3rd party companies for providing a better service.
Suffice to say I didn't install it and reverted to supervision, education and being a parent.
Wednesday, 11 December 2013
Earth and Moon from Juno
Somewhat stunning (understatment!) video made from Juno's low resolution camera during its Earth fly-by back in October:
Thanks to Phil Plait of Bad Astronomy once again...and just to think that every human that has ever lived existed in and no human has even travelled beyond the frames in that video...
Thanks to Phil Plait of Bad Astronomy once again...and just to think that every human that has ever lived existed in and no human has even travelled beyond the frames in that video...
Monday, 9 December 2013
An Academic Survey
I received this morning an email asking to me participate in a survey about "perceptions scientists have of information providers in scientific, technical and medical research fields.."
I'm always a little suspicious of such things but this does seem genuine-ish, if a little badly written; actually I was wondering if they were going to ask me to be a proxy to transfer 50 million USD from an account in the Central African Republic to somewhere on behalf of the Emporer of Botswana?
Do they really have to be so "secretive" in the wording too? I wonder what paper they are referring to, I published quite a few in 2010? What was the "careful selection criteria?"
Here's the mail:
It mentions the "Market Research Society Code of Conduct" and ensures me that my data will only be used for the primary purposes of the survey; no marketing etc...The company sending this is Confirmit, a Norwegian based company that conducts market research - just wonder why they're quoting the MRSC code of conduct presumably given that the MRSC is a UK organisation as far as I can tell.
I'd actually like to know what their privacy policy is, and how they interact with their customers...for example, do they anonymise by hashing unique identifiers? Wwhat is the full set of data that does flow to the customer? What to Confirmit keep and for how long?
Anyway, I contacted MRSC to find out whether this is genuine or not and who is actually using this data.
A few other searches suggests that this might be a request by Elsevier to find out why authors are so upset with them. I guess it isn't a secret that I frequent the category theory lists and there might be one or two there who might not be friends with Elsevier....
So, went to Elsevier's web pages to find some contact details to find out whether they were asking and again whether this is something genuine or some evil spam...clicking on "Help & Contact" gives, well....
I'll let you know how I get on if I get any replies....
I'm always a little suspicious of such things but this does seem genuine-ish, if a little badly written; actually I was wondering if they were going to ask me to be a proxy to transfer 50 million USD from an account in the Central African Republic to somewhere on behalf of the Emporer of Botswana?
Do they really have to be so "secretive" in the wording too? I wonder what paper they are referring to, I published quite a few in 2010? What was the "careful selection criteria?"
Here's the mail:
Dear Dr. Oliver ,
We are contacting you because you are the corresponding author on a paper that was published in 2010. We are conducting a survey about the perceptions scientists have of information providers in scientific, technical and medical research fields. As only a carefully selected sample of scientists and practitioners have been chosen for the study, your feedback is very valuable to us.
The survey is being conducted by a scientific, technical and medical publisher who will be revealed at the end of the survey. Under the terms of the Market Research Society Code of Conduct, it will not lead to any sales follow up and no individual (or organization) will be identified. Your results will be kept confidential and used only for research purposes.
We would be grateful if you could spare approximately 15 minutes to complete this survey. The closing date for the survey is 23rd December 2013.
To start the survey, please click ONCE on the link below (or paste it into your browser):
[link redacted]
Please do not reply to this e-mail as the inbox is not monitored. If you are having trouble with this survey you can let us know [redacted]) and we will address any technical problems as quickly as we can.
Thank you very much for your time, we really value your input.
It mentions the "Market Research Society Code of Conduct" and ensures me that my data will only be used for the primary purposes of the survey; no marketing etc...The company sending this is Confirmit, a Norwegian based company that conducts market research - just wonder why they're quoting the MRSC code of conduct presumably given that the MRSC is a UK organisation as far as I can tell.
I'd actually like to know what their privacy policy is, and how they interact with their customers...for example, do they anonymise by hashing unique identifiers? Wwhat is the full set of data that does flow to the customer? What to Confirmit keep and for how long?
Anyway, I contacted MRSC to find out whether this is genuine or not and who is actually using this data.
A few other searches suggests that this might be a request by Elsevier to find out why authors are so upset with them. I guess it isn't a secret that I frequent the category theory lists and there might be one or two there who might not be friends with Elsevier....
So, went to Elsevier's web pages to find some contact details to find out whether they were asking and again whether this is something genuine or some evil spam...clicking on "Help & Contact" gives, well....
I'll let you know how I get on if I get any replies....
Sunday, 24 November 2013
Privacy, Evidence Trails and a Change in Terminology?
One of the main aspects of personal [information] privacy is that much of the topic is that other parties would not collect nor perform any analysis of your data. The trouble is that this argument is often made in isolation, in that it somewhat assumes that the acts we perform by computer exist in a place where we can hide. For example, what someone does behind closed doors usually remains private. But, if that act is made in a public place, say, in the middle of the street by default whatever is done is not private - even if we hoped no-one saw.
Anything and everything we do on the internet is in public by default. When we perform things in public, then other people may or will see, find out and perform their own analysis to form a profile of you.
Many privacy enhancing technologies are akin to standing in the middle of a busy street and shouting "don't look!". Even if everyone looks away, more often than not there is a whole raft of other evidence to show what you've been doing.
Admittedly most of the time nobody really cares nor are actually looking in the first place. Though as it has been found out recently (and this really isn't a surprise) that some such as the NSA and GCHQ are continually watching. Even the advertisers don't really care that much; their main interest is trying to categorise you to ship a generic advertisement - and advertisers are often really easy to game...
If we really do want privacy on the internet then rather than concentrating on how to be private (or pretending that we are), we need to concentrate on how to reduce the evidence trail that we leave. Such evidence is in the form of web logs, search queries, location traces from your navigator, tweets, Facebook postings etc.
Once we have understood what crumbs of evidence is being left, we can start exploring all the side avenues where data flows (leaks) and the points where data can be extracted surreptitiously. We can also examine what data we do want released, or have no choice about.
At this moment, I don't really see a good debate about this, at least not at a technical level though there are some great tools such as Ghostery that assist in this. Certainly there is little discussion at a fundamental level which would really help us define what privacy really is.
I personally tend to take the view at the moment that privacy might even be the wrong term, or at best, somewhat a misleading term.
On the internet every detail of what we do is potentially public and can be used for good as well as evil (whatever those terms actually mean), our job as privacy professionals is to make that journey as safe as possible, hence the use of the term "information safety" to better describe what we do.
Anything and everything we do on the internet is in public by default. When we perform things in public, then other people may or will see, find out and perform their own analysis to form a profile of you.
Many privacy enhancing technologies are akin to standing in the middle of a busy street and shouting "don't look!". Even if everyone looks away, more often than not there is a whole raft of other evidence to show what you've been doing.
Admittedly most of the time nobody really cares nor are actually looking in the first place. Though as it has been found out recently (and this really isn't a surprise) that some such as the NSA and GCHQ are continually watching. Even the advertisers don't really care that much; their main interest is trying to categorise you to ship a generic advertisement - and advertisers are often really easy to game...
If we really do want privacy on the internet then rather than concentrating on how to be private (or pretending that we are), we need to concentrate on how to reduce the evidence trail that we leave. Such evidence is in the form of web logs, search queries, location traces from your navigator, tweets, Facebook postings etc.
Once we have understood what crumbs of evidence is being left, we can start exploring all the side avenues where data flows (leaks) and the points where data can be extracted surreptitiously. We can also examine what data we do want released, or have no choice about.
At this moment, I don't really see a good debate about this, at least not at a technical level though there are some great tools such as Ghostery that assist in this. Certainly there is little discussion at a fundamental level which would really help us define what privacy really is.
I personally tend to take the view at the moment that privacy might even be the wrong term, or at best, somewhat a misleading term.
On the internet every detail of what we do is potentially public and can be used for good as well as evil (whatever those terms actually mean), our job as privacy professionals is to make that journey as safe as possible, hence the use of the term "information safety" to better describe what we do.
Friday, 22 November 2013
Semiotic Analysis of a Privacy Policy
Pretty much every service has a privacy policy attached to it; these policies state the data collection, usage, purpose and expectations that the customer has to agree with before using that said service. But at another level they also attempt to signify (that's going to be a key word) that the consumer can trust the company providing the service at some level. Ok, so there's been huge amounts of press about certain social media and search service provides "abusing" this trust and so on, but we still use the services provided by those companies.
Found this paper while researching for this: Philippe Codognet's THE SEMIOTICS OF THE WEB, it starts with a quote:
So this gets me thinking, when a privacy policy is written, could we analyse that text to understand better the motives and expectations from both the customer and the service provider perspective? Effectively can we make a semiotic analysis of a privacy policy.
What would we gain by this? It is imperative that any texts of this nature portray the right image to the consumer, thus this can be used in the drafting of such a text to ensure that this this right image is correctly portrayed. For example, the oft seen statement:
"Your privacy is important to us"
is a sign in the semiotic sense, and in this case probably an 'icon' in its near universal usage. Signs are a relationship between the 'object' and 'interpretant', respectively the subject matter at hand and the clarified meaning respectively.
 |
| Pierce's Semiotic Trangle |
The object may be that we (the writer of the statement) are trying to state a matter of fact about how trust worthy we are, or at least we want to emphasise that we can be trusted.
The interpretant of this, if we are the customer, can of course vary from total trust to utter cynicism. I guess of late the latter interpretation tends to be true. Understanding the variation in interpretants is a clear method for understanding what is being conveyed by the policy itself and whether the right impression is being given to the consumer.
At a very granular level the whole policy itself is a sign and the very existance of that policy and its structure, is it long and full of legalese or short and simple? Then there's the content (as described above) which may or may not depend upon the size of the policy....as in the World's Worst Privacy Policy.
References:
Aside:
I am not sure that the web weaved by Persephone in this Orphic tale, cited in exergue of Michel Serres’ La communication , is what we are currently used to call the World Wide Web. Our computer web on the internet is nevertheless akin Persephone’s in its aims : representing and covering the entire universe. Our learned ignorance is conceiving an infinite virtual world whose center is everywhere and circumference nowhere ...Must admit, I find that very, very nice. Best I've got is getting quotes about existential crises and cosmological structures in a paper about the Semantic Web with Ora Lassila.
Tuesday, 19 November 2013
Losing Situational Awareness in Software Engineering
The cause of many aircraft accidents is
attributed to loss of situational awareness. The reasons for such a situation
are generally due to high workload, confusing or incorrect data and
misinterpretation of data by the pilot.
The common manifestation of this is mode
confusion where the pilot's expectation for a given situation (eg: approach)
does not match the actual situation. This is often seen (though not exclusively)
in highly automated environments, leading to the oft repeated anecdote:
A novice pilot will exclaim, "Hey! What's it doing now?", whereas an experienced pilot will exclaim, "Hey! It's doing that again!"
Aside: while this seems to be applied to
Airbus’ FBW models, the above can be traced back to the
American Airlines Childrenof the Magenta tutorial and specifically refers to the decidedly non-FBW Boeing
757/767 and Airbus A300 models, and not the modern computerized machines… [1]
This obviously isn't restricted to
aircraft but also to many other environments; for example, consider the
situation in a car when cruise control is deactivated due to an external
situation not perceived by the driver. Is the car slowing due to road
conditions, braking etc? Consider this in conjunction when you are expecting
the car to slow down where the deactivation has the same effect as expected;
this type of loss of situational awareness was seen in the Turkish 737 accident
at Schipol.
In an auditing environment where we obtain
a view horizontally across a company we too suffer from loss of situational
awareness. In agile, fast environments with many simultaneous projects, we
often fail to see the interactions between those projects.
Yet, understanding these links in
non-functional areas such as security and privacy is absolutely critical to a
clear and consistent application of policy and decisions.
A complicating factor we have seen is
that projects change names, components of projects and data-sets are reused
both dynamically and statically leading to identity confusion. Systems reside
locally, in the cloud and elsewhere in the Universe, terminology is stretched
to illogical extremes: big data and agile being two examples of this. Simplicity is considered a weakness and
complexity a sign of the hero developer and manager.
In systems with safetycritical properties heroes are a bad thing.
In today's so-called agile, uber-innovative,
risk-taking, fail fast, fail often, continuous deployment and development
environments we are missing the very basic and I guess old fashioned exercise
of communicating rigorously and simply what we're doing and reusing material
that already exists.
Fail often, fail fast, keep making the
same mistakes and grow that technical debt.
We need to build platforms around shared
data, not functionality overlapping, vertical components with the siloed data
mentality. This required formal communication and an emphasis on quality, not
on the quantity of rehashing buzzwords from the current zeitgeist.
In major construction projects there is
always a coordinator whose job it is to ensure that not only do individual
experts communicate (eg: the plumbers, the electricians etc) but that their
work is complimentary and that one group does not repeat the work or base that
another team has put in place.
If software engineers built a house, one
team would construct foundations by tearing down the walls that inconveniently stood
atop of already built foundations, while another would build windows and doors
while digging up the foundation as it is being constructed. A further team
charged with installing the plumbing and electrics would first endeavor to
invent copper, water and electricity...and all together losing awareness of the
overall situation.
OK, I'm being hard on my fellow software
engineers, but it is critical that we concentrate more on communication, common
goals, less "competition" [2] and ensuring that our fellow software
engineers are aware of each other's tasks.
As a final example, we will (and have!)
seen situations in big data where analytics is severely compromised because we
failed to communicate and decide upon common data models, common catalogs of
where and what data exists.
So, like the pilot faced with a situation
where he recognises he's losing his situational awareness, drop back down the
automation and resort to flying by the old-fashioned methods and raw data.
The next question is, how do you
recognise that you're losing situational awareness?
Wednesday, 13 November 2013
What does "unclassified" mean?
Previously we presented a simple security classification system consisting of four levels: secret, confidential, public and unclassified; with the latter case being the most restrictive.
While sounding counter-intuitive, a classification of "unclassified" simply means that no decision has been made. And if no decision on the actual classification has been made then it is possible that in the future that classification might be decided to be "secret". Which implies that if "unclassified" needs to be at least as strong as the highest explicit classification.
Aside: Fans of Gödel might be wondering that how can be talk about classifying something as being unclassified, which in turn is higher in some sense than the highest classification. Simply, "unclassified" is a label which is is attached to everything before the formal process of attaching on of the other labels is made.
Let's start with an example: if Alice writes a document then this document by default is unclassified. Typically that document's handling falls under the responsibility of Alice alone.
If Alice gives that document to Bob, then Bob must handle that document according to Alice's specific instructions.
By implication Alice has chosen a particular security classification. There are two choices:
We must therefore apply to a notion of authority: all documents must have an owner - this is basic document handling. That owner
While sounding counter-intuitive, a classification of "unclassified" simply means that no decision has been made. And if no decision on the actual classification has been made then it is possible that in the future that classification might be decided to be "secret". Which implies that if "unclassified" needs to be at least as strong as the highest explicit classification.
Aside: Fans of Gödel might be wondering that how can be talk about classifying something as being unclassified, which in turn is higher in some sense than the highest classification. Simply, "unclassified" is a label which is is attached to everything before the formal process of attaching on of the other labels is made.
Let's start with an example: if Alice writes a document then this document by default is unclassified. Typically that document's handling falls under the responsibility of Alice alone.
If Alice gives that document to Bob, then Bob must handle that document according to Alice's specific instructions.
By implication Alice has chosen a particular security classification. There are two choices:
- Either an explicit classification is given, eg: secret, confidential, public
- Or, no classification is given an Alice remains the authority for instructions on how to handle that document
In the latter case Alice's instructions may be tighter than the highest, explicit classification, which implies that unclassified is more restricting than, say, secret.
If Bob passes the document to Eve (or to the whole company by a reply-all) then we have a data breach. The document never implicitly becomes public through this means; though over time the document might become public knowledge but still remain officially secret. For example, if an employee of a company leaks future product specifications to the media, even though they are now effectively public, the employee (and others) who handled the data would still fall under whatever repercussions leaking secret or confidential data implies.
Still this is awkward to reconcile, so we need more structure here to understand what unclassified and the other classifications mean.
We must therefore apply to a notion of authority: all documents must have an owner - this is basic document handling. That owner
- Either assigns an explicit security classification, and all handlers of that document refer to the standard handling procedures for that security classification: referring to the security classifications standard as the authority
- Or, keeps the document as being unclassified and makes themselves the authority for rules on how to handle that document
The latter also comes with the implication that the owner of the document here is also responsible for ensuring that whatever handling rules are implied, these are consistent with the contents of the document. For example, if the document contains sensitive data then in our example, Alice is responsible for ensuring that the rules that come from her authority are as at least as strict as the highest implied security classification.
In summary, if a document or data-set is unclassified then the owner of that document is the authority deciding on what the handling rules are and that by default the rules must be at least* as strict has the highest explicit security category.
*In our classification we have the relationship:
Public < Confidential < Secret
with the statement above saying:
Public < Confidential < Secret <= Unclassified
As a final point, if Alice decides that her rules are weaker than say, confidential, but stronger than public, then it makes sense to take the next highest level as the explicit classification, ie: confidential. This way we establish the policy that all documents must eventually be explicitly classified.
Wednesday, 6 November 2013
Classifying Data Flow Processes
In previous posts (here, here, here and especially here) we presented various aspects for classifying data nominally being sent over data flows or channels between processes. Now we turn to the processes themselves and tie the classification of channels together.
Consider the data flow shown here (from the earlier article on measurement, from where we take the colour scheme of the flows) where we show the movement of, say, location information:
Notice how data flows via a process called "Anonymisation" (whatever that word actually means!) to the advertiser. During the anonymisation the location data is cleaned to a degree that business or our privacy policy allows - such boundaries are very context dependent.
This gives the first type of process, one that reduces the information content.
The other type of processes we see are those that filter data and those that combine or cross-reference data.
Filtering is where data is extracted from an opaque or more complex set of information into something simpler. For example if we feed some content, eg: a picture, then we might have processes that filter out the location data from said pictures.
Cross-referencing means that two or more channels are combined to produce a third channel containing information from the first two. A good example of this is geo-locating IP addresses which takes in as one source an IP address and another a geographical location look-up table. Consider the example below:
which combines data from secondary and primary sources to produce reports for billing, business etc.
In the above case we probably wish to investigate the whole set of processes that are actually taking place and make a considerable amount of decomposition of the process.
When combined with the classifications on the channels, particularly information and security classes we can make some substantial reasoning. For example, if there is a mismatch in information content and/or security classifications then we have problems; similarly if some of these are transported over insecure media.
To summarise, in earlier articles we explained how data itself may be classified, and here how processes may be classified according to a simple scheme:
In a later article I'll go more into the decomposition and refinement of channels and processes.
Consider the data flow shown here (from the earlier article on measurement, from where we take the colour scheme of the flows) where we show the movement of, say, location information:
Notice how data flows via a process called "Anonymisation" (whatever that word actually means!) to the advertiser. During the anonymisation the location data is cleaned to a degree that business or our privacy policy allows - such boundaries are very context dependent.
This gives the first type of process, one that reduces the information content.
The other type of processes we see are those that filter data and those that combine or cross-reference data.
Filtering is where data is extracted from an opaque or more complex set of information into something simpler. For example if we feed some content, eg: a picture, then we might have processes that filter out the location data from said pictures.
Cross-referencing means that two or more channels are combined to produce a third channel containing information from the first two. A good example of this is geo-locating IP addresses which takes in as one source an IP address and another a geographical location look-up table. Consider the example below:
which combines data from secondary and primary sources to produce reports for billing, business etc.
In the above case we probably wish to investigate the whole set of processes that are actually taking place and make a considerable amount of decomposition of the process.
When combined with the classifications on the channels, particularly information and security classes we can make some substantial reasoning. For example, if there is a mismatch in information content and/or security classifications then we have problems; similarly if some of these are transported over insecure media.
To summarise, in earlier articles we explained how data itself may be classified, and here how processes may be classified according to a simple scheme:
- Anonymising
- Filtering
- Cross-Referencing
In a later article I'll go more into the decomposition and refinement of channels and processes.
Friday, 1 November 2013
Measurement and Metrics for Information Privacy
We have already discussed an analogy for understanding and implicitly measuring the information content over a data channel. The idea that information is an "infectious agent" is quite a powerful analogy in the sense that it allows us better to understand the consequences of information processing and the distribution of that data better, viz:
Ley us classify information content into seven basic categories: financial, health, location, personal, time, identifiers and content. Just knowing what kinds of data are present as we have already discussed gives us a mechanism to pinpoint where more investigation is required.
We can then go further an pick out particular data flows for further investigation and then map this to some metric of contamination:
For example, transporting location data has a particular set of concerns, enough to make any privacy professional nervous at the least! However if we examine a particular data flow or store we can evaluate what is really happening, for example, transmitting country level data is a lot less invasive than highly accurate latitude and longitude.
Now one might ask why not deal with the accurate data initially? The reasons are that we might not have access to that accurate, field-level data, we might not want to deal with the specifics at a given point in time, specific design decisions might not have been made etc.
Furthermore, for each of the seven categories we can give some "average" weighting and abstract away from specific details which might just complicate any discussion.
Because we have a measure, we can calculate and compare over that measurement. For example, if we have a data channel carrying a number of identifiers (eg: IP, DeviceID, UserID) we can take the maximum of these as being indicative of the sensitivity of the whole channel for that aspect.
We can compare two channels, or two design decisions, for example, a channel carrying an applicationID is less sensitive (or contaminated) than one carrying device identifiers.
We also can construct a vector over the whole channel composed out of the seven dimensionsb above to give a further way of comparing and reasoning about the level of contamination or sensitivity:
We've only spoken about a single data flow at the moment, however the typical scenario is for reasoning over longer flows, for example, we might have our infrastructure set up as below*
In this example we might look at all the instances where AppID and Location are specified together and use a colour coding such that:
There are a number of points we have not yet discussed, such as that some kinds of data can be transformed into a different type. For example some kinds of content such as pictures inherently contain device identifiers, locations etc. Indeed the weighting for some categories such as content might be very much higher than that of identifiers for example - unless the investigation is made. Indeed it does become almost a trivial exercise for some to explicitly hide sensitive information inside opaque data such as images and not declare then when a privacy audit is made.
To summarise, we've a simple mechanism for evaluating information content over a system in quantative terms, complete with a refinement mechanism that allows us to work at many levels of abstraction depending upon situation and context. Indeed, what we are doing is explcitly formalising and externalising our skills when performing such evaluations, and through analogies such as "infection control" providing means for easing the education of professionals outside of the privacy area.
*This is not indicative of any system real, living or dead, but just an example configuration.
- Any body of data which is containing certain amounts and kinds of sensitive data we can consider to be non-sterile
- Information which is truly anonymous is sterile
- Mixing two sets of information produces a single set of new information which is as at least as unclean as the dirtiest set of data mixed, and usually more so!
- The higher the security classification the dirtier the information
Ley us classify information content into seven basic categories: financial, health, location, personal, time, identifiers and content. Just knowing what kinds of data are present as we have already discussed gives us a mechanism to pinpoint where more investigation is required.
We can then go further an pick out particular data flows for further investigation and then map this to some metric of contamination:
For example, transporting location data has a particular set of concerns, enough to make any privacy professional nervous at the least! However if we examine a particular data flow or store we can evaluate what is really happening, for example, transmitting country level data is a lot less invasive than highly accurate latitude and longitude.
Now one might ask why not deal with the accurate data initially? The reasons are that we might not have access to that accurate, field-level data, we might not want to deal with the specifics at a given point in time, specific design decisions might not have been made etc.
Furthermore, for each of the seven categories we can give some "average" weighting and abstract away from specific details which might just complicate any discussion.
Because we have a measure, we can calculate and compare over that measurement. For example, if we have a data channel carrying a number of identifiers (eg: IP, DeviceID, UserID) we can take the maximum of these as being indicative of the sensitivity of the whole channel for that aspect.
We can compare two channels, or two design decisions, for example, a channel carrying an applicationID is less sensitive (or contaminated) than one carrying device identifiers.
We also can construct a vector over the whole channel composed out of the seven dimensionsb above to give a further way of comparing and reasoning about the level of contamination or sensitivity:
| (FIN=0,HLT=0,LOC=10,PER=8,TIM=3,ID=7,CONT=0) |for some numerical values gien to each category. Arriving at these values will be specific to a given wider context and then the weighting given to each, but there is one measure which can be used to ground all this, and that is of information entropy, or, how identifiying the contents are to a given, unique human being. A great example of this is given at the EFF's Panopticlick pages.
<
| (FIN=3,HLT=2,LOC=4,PER=4,TIM=2,ID=9,CONT=2) |
We've only spoken about a single data flow at the moment, however the typical scenario is for reasoning over longer flows, for example, we might have our infrastructure set up as below*
- Black: unknown/irrelevant
- Red: high degree of contamination, both AppID and Location unhashed and accurate respectively
- Yellow: some degree of contamination, AppID may be hashed(+salt?) or Location at city level or better
- Green: AppID randomised over time, hashed, salted and Location at country level or better
There are a number of points we have not yet discussed, such as that some kinds of data can be transformed into a different type. For example some kinds of content such as pictures inherently contain device identifiers, locations etc. Indeed the weighting for some categories such as content might be very much higher than that of identifiers for example - unless the investigation is made. Indeed it does become almost a trivial exercise for some to explicitly hide sensitive information inside opaque data such as images and not declare then when a privacy audit is made.
To summarise, we've a simple mechanism for evaluating information content over a system in quantative terms, complete with a refinement mechanism that allows us to work at many levels of abstraction depending upon situation and context. Indeed, what we are doing is explcitly formalising and externalising our skills when performing such evaluations, and through analogies such as "infection control" providing means for easing the education of professionals outside of the privacy area.
*This is not indicative of any system real, living or dead, but just an example configuration.
Subscribe to:
Posts (Atom)