2017 in Tech...

Two posts in one day - that's a good start so far.

Here's User Friendly's take on 2017 in Technology ... this one is a bit deep :-)

Seminar: Software as a Medical Device

Seminar: Software as a Medical Device:

Safety and security. 

January 5, 9-11 am 

Teaduspargi 6/1, Tallinm, Estonia

Seminar room: Merkuur 

Connected Health cluster presents a practical seminar to help health IT developers and startups plan and manage smoothly their products to comply with needed standards and rules.

9:00 What is a software as a medical device and what is required to get regulatory compliant products on the market - overview of medical device software safety, regulations in EU and US, standards and FDA guidance - Dr. Marion Lepmets, Co-Founder & CEO of SoftComply – 30 min presentation + 15 min Q&A

9:45 Privacy Engineering and Health Data: IT and IoT - Dr. Ian Oliver, Security Specialist at Bell Labs – 30 min presentation + 15 min Q&A

10:30 Discussion and 1-2-1 Q&A

Please register by January 3 the latest: services@tehnopol.ee 

Free for Science Park Tehnopol network and service clients and Connected Health cluster members. 30€ + vat for others.

Privacy, Evidence Trails and a Change in Terminology?

One of the main aspects of personal [information] privacy is that much of the topic is that other parties would not collect nor perform any analysis of your data. The trouble is that this argument is often made in isolation, in that it somewhat assumes that the acts we perform by computer exist in a place where we can hide. For example, what someone does behind closed doors usually remains private. But, if that act is made in a public place, say, in the middle of the street by default whatever is done is not private - even if we hoped no-one saw.

Anything and everything we do on the internet is in public by default. When we perform things in public, then other people may or will see, find out and perform their own analysis to form a profile of you.

Many privacy enhancing technologies are akin to standing in the middle of a busy street and shouting "don't look!". Even if everyone looks away, more often than not there is a whole raft of other evidence to show what you've been doing.

Admittedly most of the time nobody really cares nor are actually looking in the first place. Though as it has been found out recently (and this really isn't a surprise) that some such as the NSA and GCHQ are continually watching. Even the advertisers don't really care that much; their main interest is trying to categorise you to ship a generic advertisement - and advertisers are often really easy to game...

If we really do want privacy on the internet then rather than concentrating on how to be private (or pretending that we are), we need to concentrate on how to reduce the evidence trail that we leave. Such evidence is in the form of web logs, search queries, location traces from your navigator, tweets, Facebook postings etc.

Once we have understood what crumbs of evidence is being left, we can start exploring all the side avenues where data flows (leaks) and the points where data can be extracted surreptitiously. We can also examine what data we do want released, or have no choice about.

At this moment, I don't really see a good debate about this, at least not at a technical level though there are some great tools such as Ghostery that assist in this. Certainly there is little discussion at a fundamental level which would really help us define what privacy really is.

I personally tend to take the view at the moment that privacy might even be the wrong term, or at best, somewhat a misleading term.

On the internet every detail of what we do is potentially public and can be used for good as well as evil (whatever those terms actually mean), our job as privacy professionals is to make that journey as safe as possible, hence the use of the term "information safety" to better describe what we do.

Systems Safety - Defining Moments

As I've been concentrating on "safety improvements", or at least techniques for the improvement of system I've tended to concentrate on four areas:

• Aviation
• Industrial
• Medical (specifically surgical)
• Software Engineering (specifically information privacy)

The parallels between theses areas when it comes to what each define as "safety" and the techniques should be obvious. However the question remains what actually triggered each of these respective areas to take a more inherent safety approach and then what will trigger a similar approach with regards to information safety?

 

Above Diagram Key: Y-Axis: relative degree of safety embedded into that discipline, X-Axis, year of time since seminal incident.

Aviation safety's seminal moment was the 1935 crash of a Boeing Model 299 aircraft during a presentation flight. Instead of blaming the pilots, effort was made to understand the causes of the accident and develop techniques to help prevent similar accidents in the future.

For industrial safety the seminal moment was the 1974 Flixborough Disaster in the UK. This resulted in work on the design of industrial plants and the development of the notion of "inherent safety".

Surgical safety has quite a long tradition especially with the development of anaesthetic safety from the 1960s and the introduction of a proper systems approach. However anesthetists seem not to feature prominently as surgeons and doctors so the fame would probably go to Peter Pronovost et.al. for the Central Line checklist. This was probably one of the major contributors to the WHO Surgical Safety Checklist discussed in detail in Atul Gawande's book The Checklist Manifesto which brings together much of the above incidents. 

If you're still in doubt maybe Atul Gawande's article in the New Yorker magazine entitled The Checklist: If something so simple can transform intensive care, what else can it do? (Dec 10, 2007) might help.

Getting back to the crux of this article, what is the incident that will cause the wholesale change in attitudes and techniques to software engineering that instills such a sense of discipline that we can eradicate errors to such a degree that we could compare ourselves favourably with other disciplines?

The increasingly frequent hacking and information leaks? The NSA wiretapping and mass surveillance? Facebook and Google's privacy policies? None of these have had any lasting effect upon the very core of software engineering if any at all. Which either means that we place such low value on the safety of our information or that the economics of software are so badly formulated in society that the catastrophe would have to be so huge that it would have to cause societal change?

Interestingly, in software engineering and computer science we're certainly not short on techniques for improving the quality and reliability of the systems we're developing: formal methods (eg: Alloy, B, Z, VDM etc), proof, simulation, testing, modelling (in general). What we probably lack is the simplicity of a checklist to guide us through the morass of problems we encounter. In this last respect, this is why I think we're more like surgeons that modern day aviators; or, maybe some of us are like the investigators to the 1935 Boeing crash and other aviation heroes learning their trade?

What is the economic component of privacy to the end-user?

Privacy is framed in terms of ethical, moral and economic arguments. Certainly most research supports that nearly everyone is worried about their on-line privacy, yet rarely if at all we've seen a company actually fail because of their attitude to privacy. Some companies have changed their ways admittedly, but the change has been minimal and after the initial fuss it is very much business as usual. Privacy seems not to be much of an ethical or moral issue to most people but rather an emotional one.

Given the above we don't see much of an economic argument either, at least not from the end-user side of things. The economic argument for using certain social networking services, sharing personal photographs, details etc seems to outweigh the potential disadvantages - which tend to be worries about to where data is being shared or sold, or who might be viewing that material.

Privacy is rarely framed in terms of an economic argument beyond the "data mart" idea but rather in emotional terms: nothing to hide, nothing to fear perhaps? Privacy is lost in the security and liberty arguments too being equated with concealment and freedom. Consider the argument put forward by US congressman Paul Rand [1] on privacy.

Laws are changing to emphasise the economic argument to the companies that provide internet services and there we do see a strong argument for a more robust privacy function. Whether we on the business side of things are addressing privacy in the right manner and in the same economic terms as our end-users is another matter altogether. For companies the economic argument for developing and adhering to an information safety (privacy) is very clear. We have a good economic argument for treating end-users' information with respect...whatever that is.

For the end-user the economic argument is unclear. Why should an end-user choose one service provider over another? What are the terms of this economic argument and have they even been defined at all?

References

[1]  Senator Rand Paul Talks Tech, Civil Liberties, and Keeping the Government Out of Your Email. By Spencer Ackerman 05.30.13 Wired

More Maxims for Privacy

I wrote some maxims for privacy a while back, but still I'm looking for that crystalisation of privacy - the "one slider" or "one sentence" that encompasses the foundations, or the basics of privacy.

For the most part privacy comes down to data collection and subsequent usage of that data - the rest is just additions to that. At least if we're concentrating on privacy and not the wider scheme of information management of which I believe privacy is just a sub-speciality, albeit a rather important one.

So, when dealing with information:

"If you don't have a use for it, 
don't collect it!"

For me this sums it up. It links data collection and usage in a way that clearly states that just collect what you need right now; don't even think about the future uses yet. Interestingly I also think that deliberately restricting the data collection right at the start to what you absolutely know you are going to use immediately forces you (the software/system developer) to better focus on the product at hand - "Slow Data" anyone?

In these agile development days however, it is often argued that we'll develop the usages later and collect everything now. How often does this really happen? And, if you really were agile then you'd construct your system initially to do the minimum it needs to and get that out to the customer for their appraisal. If that goes well (or not), then modify as necessary during later stages in your agile development process.

If you're not agile (which apparently is waterfall, though I'm not sure), then you should have worked out what you need and work from there; which again should be self-limiting on the data collection. Surely in a good, fully worked out design you wouldn't be collecting superfluous things?

So, that's it, the essence of privacy as a single sentence; the rest is just layers pertaining to things such as provenance, data retention, purpose, infrastructure etc - that's what makes good information management a whole discipline in itself, of which privacy is one small, but important part.

Code is Law, Inherent Privacy and a Few Uncomfortable Issues

Lawrence Lessig stated that "code is law" - a maxim that above all should be the most critical in software engineering, especially when put in the context of implementing privacy and security.

I want to talk about some issues that worry me slightly (ok, a lot!). The first is that despite of policies, laws etc, the final implementation of anything related with privacy is in the code the programmers write. The second is that we are building our compliance programmes upon grand schemes and policies and paying piecemeal attention to the actual act of software engineering. The latter we attempt to wrap up in processes and "big ideas", for example, Privacy by Design.

Now before the PbD people get too upset, there's nothing wrong with stating and enumerating your principles, the Agile Manifesto is a great example of this, however there is no doubt that many implementation of agile are poor at best and grossly negligent and destructive at worst. The term used is "technical debt".

Aside: the best people I've seen conduct software development in an agile manner are formal methods people...I guess due to the discipline and training in the fundamentals they've received. This also applied to experienced architects, engineers and programmers for whom much of this formality is second nature.

Addressing the first point: no matter how many policies or great consumer advocates or promises you make, at the end of the day, privacy must be engineered into the architecture, design and code of your systems. It does not matter many powerpoint slides or policy documents or webpages your write, unless the programmers "get it", you can forget privacy, period!

Aside: Banning powerpoint may not be such a bad idea....

Herein lies a problem, the very nature of privacy in your systems means that it crosscuts every aspects of your design and ultimately your whole information strategy. Most of these things do not obviously manifest themselves in the design and code of your systems.

To solve this there must be a fundamental shift from the consumer advocacy-legal focus of privacy to a much deeper, technical or engineering, even scientific approach. This however does not just mean focusing on the design and code, though that is fundamental to the implementation, but to the whole stack of management and strategy from the highest directors to the programmers.

I've seen efforts in this direction but stop at the product management - "Hey, here are the privacy requirements - implement them!" ... which does feel good in that you are interacting, or believe that you are interacting, with the products you are producing but still not sufficiently with the people who really build these. Just producing requirements doesn't help: you need that interaction and communication right across the company.

Of course all of the above is extremely difficult and leads us to our next point which is how we build our compliance programmes in the first place. The simple question here is "are you fully inclusive?", meaning do you include programmers, architects (technical people with everyday experience) or is the programme run by non-technical, or formerly technical staff? Invariably it is the latter.

Compliance programmes must be inclusive otherwise the necessary inherency required to successfully and sufficiently implement the ideas and strategies of that programme will be lost - usually in a sea of powerpoint and policy documents.

Firstly in order to achieve inherent privacy (or security, or xyz) focus must lie on onboarding and educating the programmers, the designers and the architects and less focus on the management, prescription and consumer advocacy. Secondly, any compliance programme must be inclusive and understand the needs of the said technical staff. Thirdly, the engineering and technical staff are the most critical components in your organisation.

Compliance programmes are often measured on the amount of documentation produced (number of slides even?), however this ends up with a self feeding process where for the compliance programme to survive it needs to keep the fear of non-compliance at the fore. Read Jeff Jarvis' article on Privacy Inc.: Scare and Sell and then Jim Adler's talk about PII2012 on The Emergent Privacy-Industrial Complex and you get an idea of what is going wrong. Avoid at all costs creating a privacy priesthood in your compliance programmes.

Aside: This might be good old fashioned economics - if a compliance programme actually worked then there'd be no need for the programme in the end.

There are two interrelated caveats that also need to be discussed, the first of which is that any work in privacy will expose the flaws, holes and crosscutting issues across your products, development programmes and management and engineering skill bases. For example, a request to change a well crafted design to cope with some misunderstood ambiguity in privacy policy is going to end in tears for all concerned. It will demand of management, engineering and your compliance programme a much deeper [scientific] knowledge of what information your products are using, carrying, collecting and processing - to a degree uncommonly found in current practices.

The fundamental knowledge require to really appreciate information management and privacy is extensive and complex. Awareness courses are a start but I've seen precious few courses even attempting to cover the subject of privacy from a technical perspective.

Secondly, privacy will force you to examine your information strategy - or even create and information strategy - and ask very awkward and uncomfortable questions about your products and goals.

Measuring Privacy against Effort to Break Security

As part of my job I've needed to look at metrics and measurement of privacy. Typically I've focussed on information entropy versus, say, number of records (define "record") or other measurements such as amount of data which do not take into consideration the amount of information, that is, the content of the data being revealed.

So this lead to an interesting discussion* with some of my colleagues where we looked at a graph like this.

The y-axis is a measure of information content (ostensibly information entropy wrt to some model) and the x-axis a measure of the amount of force required to obtain that information. For any given hacking technique we can deliniate a region on the x-axis which corresponds to the amount of sophistication or effort placed into that attack. The use of the terms, effort and force here come from the physics and I think we even have some ideas on how the dimensions of these map to the security world, or actually what these dimensions might be.

So for a given attack 'x', for example an SQL inject attack against some system to reveal some information 'M', we require a certain amount of effort just for the attack to reveal something. If we make a very sophisticated attack then we potentially reveal more. This is expressed as the width of the red bar in the above graph.

One conclusion here is that security people try to push the attack further to the right and even widen it, while privacy people try to lower and flatten the curve, especially through the attack segment.

Now it can be argued that even with a simple attack, over time the amount of information increases, which brings us to a second graph which takes this into consideration:

Ignoring the bad powerpoint+visio 3D rending, we've just added a time scale (z-axis, future towards back), we can now capture or at least visualise the statement above that even an unsophisticated attack over time can reveal a lot of information. Then there's a trade-off between a quick sophisticated attack versus a long, unsophisticated attempt.

Of course a lot of this depends upon having good metrics and good measurement in the first place and that we do have real difficulties with, though there is some pretty interesting literature [1,2] on the subject and in the case of privacy some very interesting calculations that can be performed over the data such as k-anonymity an l-diversity.

I have a suspicion that we should start looking at privacy and security metrics from the dimensional analysis point of view and somewhat reverse engineer what the actual units and thus measurements are going to be. Something to consider here is that the amount of effort or force of an attack is not necessarily related to the amount of computing power, for example, brute forcing an attack on a hash function is not as forcible as a well planned hoax email and a little social engineering.

If anyone has ideas on this please let me know.


References

[1] Michele Bezzi (2010) An information theoretic approach for privacy metrics. Transactions on Data Privacy 3, pp:199-215
[2] Reijo M. Savola (2010) Towards a Risk-Drive Methodology for Priavcy Metrics Development. IEEE International conference on Social Computing/IEEE International Conference on Privacy, Security, Risk and Trust.



*for "discussion" read 'animated and heated arguments, a fury of writing on whiteboards, excursions to dig out academic papers, mathematics, coffee etc' - all great stuff :-)

Teaching Privacy

It often surprises me that many of the people advocating privacy don't actually understand the things that they're trying to keep private, specifically information. Indeed the terms data and information are used interchangeably and there is often little understanding of the actual nature and semantics of said, data and information.

I've run courses on data modelling, formal methods, systems design, semantics and now privacy - the latter however always seems to be "a taster or privacy" or "brief introduction to privacy" and there rarely is the chance to get into specifics about what information is.

This of course has some serious implications and one of the best I can find is when we talk about anonymisation. I've seen horrors such as statements "if you hash this identifier, then it is anonymous" or "if we randomise this data then we can't track" or lately, "if we set this flag to '1' then no-one will track you anymore". In the first case I refer people back to the AOL Data Leak and the dangers of fingerprinting, semantic analysis and simple cross-referencing.

I made a study a while back based on the leak of 16,000 names from various Finnish education organisations (plus maybe other places). It was very interesting to see that even with the released list that contained dates of birth and last names how many were already unique, and even in the cases where there existed common Finnish names how easy it was to trace these back to a unique person. Actually going to the next step and verifying this with that person would I guess have been somewhat illegal or if not, unethical to say the least. Social engineering would have been very easy in many of these cases I'm sure.

So given cases like these and the current dearth of educational material I though it would be nice to try to put together a more comprehensive and deeper set of material. Some universities are already doing this and there also exist industrial qualifications such as those by the IAPP, however at this stage all ideas are welcome.

Now I want to specifically address a technical audience: software engineers, computer scientists - the people who end up building these systems because that's where I feel much breaks down - for many reasons but I won't appoint blame here - that's not really constructive in the current context.

First of all I want to break things down into 3 logical segments, actually there are 4 but I'll discuss that one later:

• Legal
• Consumer Advocacy
• Technical

 and address each area individually.

Legal is relatively straightforward in that an understanding of principles of privacy, how various jurisdictions view data, information, anonymisation, cross-referencing, children and minors, cross-border data transfer, retention and data collection and a discussion of certain practices, eg: EU, US, China, India etc. This discussion doesn't have to be heavy but an understanding of what the law states and how it interprets things is critical. Also from here we should get an understanding of how the law affects the engineering side of things: common terminology as a good example.

Consumer advocacy is really the overview material in my opinion - what are the principles of privacy, for example Cavoukian's Privacy by Design as an example (even if I'm not happy with the implementation of these), how to consumers view privacy, what is the reality (say vs do) and also various case studies such as how consumers view Google, Apple, Nokia, Facebook, various Governments, technologies such as NFC, mobile devices, 'Smart Televisions', direct marketing and advertising, store cards etc. Out of this comes an understanding of how privacy is viewed and even an appreciation of why we don't get privacy: anti-privacy if you like.

The technical aspect takes in many technologies, rather than describe, I'll list them (and this will be non-exhaustive and in no particular order)

• Basic Security - Web, Encryption, Hashing, Hacking (XSS etc), authentication (OpenID, OAuth etc), differences/commonalities between privacy and security, mapping privacy problems into security problems as a solution
• Databases - technologies, design, schema development (eg: relational theory), "schema-less" databases, cross-referencing, semantic isolation
• Semantics - ontologies, classifications, aspect, Semantic Web
• Data-flow
• Distributed Systems - networking and infrastructure
• API design - browsers, apps, web-interfaces, REST
• Data Collection - primary vs secondary vs infrastructure, logging
• Policy - policy languages, logic, rules, data filtering
• Anonymisation - data cleansing
• Identifiers - tracking, "Do Not Track"
• User-Interface
• Metrics for privacy - entropy
• Information Types and Classification - location, personally identifiable information, identifiers, PCI, health/medical data

as you can see the list is extensive and an understanding of each of these areas is critical to building systems that honour and preserve privacy in its various forms (as described in the consumer advocacy and legal sections). The main point here is to provide software engineers and computer scientists with the tools to implement privacy in a meaningful manner.

Now that we have outlined the three areas we can look at the fourth which binds these together and which I tentatively call "Theory of Privacy".

Obviously something binds these areas together and there does exist a huge body of work on the nature of information and its classifications. I particularly like the approach by Barwise and Seligman in the 1997 book Information Flow: The Logic of Distributed Systems*. I believe we can quite easily get into all sorts of interesting ontology, semantics and even semiotic discussions. Shannon's Information Theory and notions of entropy (eg: Volkstein's book: Entropy and Information) are fundamental to many things. I think this really is an area that needs to be opened up and addressed seriously and anything that binds together and provides a common language to unify consumer advocacy, the law and software engineering is critical.

Finally, no outline of a course would be complete with some preliminary requirements and a book list. For the former an understanding of computer systems and basic computer security is a must (there is no privacy without security), a grounding in software engineering techniques and a dose of computer science similarly. For the books, my first draft list would include:

• Barwise, Seligman. Information Flow
• O'Hara, Shadbolt. The Spy in the Coffee Machine: The End of Privacy as We Know It
• Solove. Understanding Privacy
• Nissenbaum. Privacy in Content: Technology, Policy, and the Integrity of Social Life
• Solove: The Future of Reputation: Gossip, Rumour, and Privacy on the Internet

*somebody should make a movie of this.

Opa Language Tutorial: Part 2

Continuing from part 1, I want to extend my application to actually work with some data. Let's go back to the requirements from earlier and look at how we wish to work with those and map these into the REST idea.

Firstly we require a list of regex expressions, with identifiers and descriptions [1], for example:

• Expression visa1 describes the card number on Visa cards: ^4[0-9]{12}(?:[0-9]{3})?$
• Expression master1 describes the card number on Mastercard cards: ^5[1-5][0-9]{14}$
• Expression diners1 describes the card number on Diners Card cards: ^3(?:0[0-5]|[68][0-9])[0-9]{11}$

So we have a data structure of the form: identifier x regex x description, which in turn can be all represented using strings. Opa allows us to define such data structures as types:

type regexExpression = { string exprID, string regex, string description };

As we're being formal about this, let us also decide that the field exprID should be unique, ie: a primary key for referring to any given expression.

Secondly we want to store these somewhere and conveniently Opa provides us with a database mechanism. These are constructed similarly to types above but use the database keyword and database paths:

database regexDB {

   stringmap(regexExpression) /expressions

} 

The above code defines a database called regexDB which contains a "stringmap" structure of regexExpressions and identified using the database path /expressions.

Firstly the fully qualified database path is /regexDB/expressions . Each path can be considered to be a field in the database. Some "fields" we would rather act more like tables and so we use the Opa type stringmap which is equivalent to a dictionary or hashtable in other languages. Stringmap defines the keys to be of type string and we define the value for those keys to be - in this case - regexExpressions.

Part of the REST idea was based upon the CRUD concept and indeed all well behaved databases should follow the CRUD concepts. Fortunately REST and CRUD do map quite nicely together and we can define the operations upon the database and data type we have defined:

First we have to note that the /expressions database path can either be used as is, that is there is a resource, eg: http://a.b.c.d/expressions that we can navigate to, or, because we're introduced the concept of a key through the stringmap we could refer directly to resources contained in or under, eg: http://a.b.c.d/expressions/visa1. This gives us the following table:

Verb	/expressions	/expressions/"k"
GET	return a list of expression identifiers even if empty. Return a 200 success code	return the full details of expression "k". If "k" doesn't exist then return a 404 error
POST	add the requested object if the supplied key in the object doesn't exist. Return a 201 success code as well as the key "k", otherwise return a 400 error.	not allowed, return a 400 error
PUT	not allowed, return a 400 error	modify the database with the given object if the supplied key both exists in the database and matches the key in the supplied object - return a 200 success code. In all other circumstances return a 400 error.
DELETE	not allowed, return a 400 error	delete the database entry with the given key if it exists in the database and return a 200 success code. In all other circumstances return a 400 error.

Let us look at the above as a check that they are a) consistent/correct and b) satisfy the various REST principles or invariants. For the GET operation the key must exist otherwise error, similarly for PUT and DELETE. For POST the key must not exist initially. In terms of REST each GET is certainly idempotent and POST, PUT and DELETE act according if accessed multiple times - this all sounds good. We should really write a much better formal specification of this, however we can leave that to the reader :-) So that's a lot of specification and now we should move on to the fun bit of programming! We now define a REST endpoint for the above and this is achieved by adding an additional case statement to our dispatcher:

function start(url) {

   match (url) {

       case {path: [] ... }: hello();

       case {path: ["expressions" | restofpath] ...} : expressionsRESTendpoint(restofpath);

       case {~path ...}: error();

   }

}

and also write some skeleton code for our new expressionsRESTendpoint() function:

function expressionsRESTendpoint(path) {

   Debug.warning("Path is {path}.");

   match(HttpRequest.get_method()){

      case{some: method}:

          Debug.warning("Method is {method}.");

          Resource.raw_status({success});

      default:

          Resource.raw_status({bad_request});

   }

}

The match statement in the start() function should be reasonably self-explanatory in that we call expressionsRESTendpoint() when we match the path of the URI such that the first entry is the "expressions". Note this is a list comprehension so that the variable "restofpath" now refers to the remainder of the list.

The skeleton code for expressionsRESTendpoint() takes a list as its parameter. The first line of this function is a debugging statement - this is Opa's way of debugging by printf: Debug.warning("xyz"). At run-time these statements are output by generated executable to the terminal (stderr?). This statement also shows a feature of Opa which is that you can enter a code block inside a string (or xhtml as we shall see later) - the use of { and } braces denotes the code block. Remember Opa is functional so that a code block will return a value, in this case it is the list in the path variable. Opa also makes the type conversion to string implicitly.

In this function we have a match statement which takes the current context of the HttpRequest to the server and extracts the HTTP method from it. This function call returns an "option"-type in Opa parlance so we need to check whether it actually does have a value or not. For Haskell programmers the equivalent is Maybe. They way we do this is to construct yet another match statement (pattern matching is your friend) and choose between whether the value exists or not. Opa uses the keyword some to find out whether this value exists or not, an in the above, assigns the value to the variable named "method".

Aside: I really need to write a proper tutorial for how match-case and pattern matching works.

Opa's match-case statements also can have a default case to catch cases where nothing is matched. A good tip is always to include a default statement, even if you think you've managed to catch 100% of cases; defensive programming!

Again we include a debugging statement to return the name of the http method. After this we generate a Resource object but this time just built so that it is just effectively an http return code - in this case HTTP1.1 200 OK or success. The default case is similar except that we return the 400 Bad Request error.

Now we can test the above code - after compiling it. To test we use curl as a way of easily inspecting the return codes and controlling what we send. If we issue the following commands:

ian@U11-VirtualBox:~/opatutorial$ curl -X GET http://127.0.0.1:8080/expressions
ian@U11-VirtualBox:~/opatutorial$ curl -X PUT http://127.0.0.1:8080/expressions/ford
ian@U11-VirtualBox:~/opatutorial$ curl -X PUT http://127.0.0.1:8080/expressions/ford/prefect

then on the terminal running our JavaScript executable (remember to compile and run!) then we'll see the output:

[Opa] Server dispatch Decoded URL to /expressions
[Opa] Debug Path is [].
[Opa] Debug Method is {get = {}}.
[Opa] Server dispatch Decoded URL to /expressions/ford
[Opa] Debug Path is [ford].
[Opa] Debug Method is {put = {}}.
[Opa] Server dispatch Decoded URL to /expressions/ford/prefect
[Opa] Debug Path is [ford,prefect].
[Opa] Debug Method is {put = {}}.         

Each group of 3 lines corresponding to the individual curl commands issued. Opa reports the dispatch anyway, the rest is from our Debug.warning commands. Note the path provided and the Path output by the debugging command as a list from the "restofpath" variable set during the match-case pattern matching.

Ok, so what we have is out REST end-point working and tested. Let's finish this off with updating the skeleton code we've written to match the http methods and write code to input the data sent to our database; for the moment we'll just implement POST and get the database running and populated.

function expressionsRESTendpoint(path) {
   Debug.warning("Path is {path}.");
   match(HttpRequest.get_method()) {     
      case{some: method}:
         match(method) {
             case{get}:
                Debug.warning("GET method called");
                Resource.raw_status({success});
             case{post}:
                expressionsPost();    
             default:
                Debug.warning("Some other method called {method}");
                Resource.raw_status({method_not_allowed});           
         }
      default:
          Resource.raw_status({bad_request});
   }        
}


and add a method to handle the post: expressionsPost(). The path isn't interesting for the post so we don't pass it. Out new function however takes the form:

function expressionsPost() {
   Debug.warning("Body is {HttpRequest.get_body()}");
    match(Json.deserialize(HttpRequest.get_body() ? "")) {
       case{some: jsonobject}:
          match(OpaSerialize.Json.unserialize_unsorted(jsonobject)) {
             case{some: regexExpression e}:
                Debug.warning("got object with fields {e.exprID}, {e.regex}, {e.description }");//                /regexDB/expressions[e.exprID] <- e;
                Debug.warning("record loaded into database...try a get now");
                Resource.raw_status({success});
             default:
                Debug.warning("missing fields in JSON");
                Resource.raw_status({method_not_allowed});
          }
       default:
          Debug.warning("something failed");
          Resource.raw_status({bad_request});
    }
}

Now before anyone asks, yes the above does not conform to my spec (part 3 anyone?) and its full of debug stuff and a commented line referring to the database. The Debug.warning statements you can guess. The get_body() method returns the body of the call to the server similarly to get_method() used earlier.

I will now make an implementation choice: all our REST calls take their parameters as a JSON object in the body of the message. This will make the next line mroe understandable.

HttpRequest.get_body() ? ""

The ? operator is a short-cut for dealing with option types (explained earlier). If in this case a body does exist then return that, else return an empty string. The result of this is passed to the function Json.deserialize will turns whatever string is in the HttpRequest body into JSON. The following case statement:

case{some: jsonobject}

pattern matches that if we have a some JSON object then return this in the variable jsonobject, if not then this is caught by the default: clause later.

We perform a similar trick in the next match statement as shown in this code snippet:

match(OpaSerialize.Json.unserialize_unsorted(jsonobject)) {
  case{some: regexExpression e}:
    Debug.warning("got object with fields {e.exprID}, {e.regex}, {e.description }");

     ...

where

OpaSerialise.Json.unserialize_unsorted(jsonobject)

turns our valid JSON into hopefully valid Opa...if successful then we get some Opa which we coerce (type cast) into our type regexExpression and return it in variable "e". The Debug.warning statement should be self-explanatory.

Aside: yes, this could have been written neater and there might be a better way of dealing with unserialising JSON...I'd certainly like to know the latter.

If we run the above code, say with the two requests:

ian@U11-VirtualBox:~/opatutorial$ curl -X POST -d "hello" http://127.0.0.1:8080/expressions
ian@U11-VirtualBox:~/opatutorial$ curl -X POST -T "regex1" http://127.0.0.1:8080/expressions

NB: regex1 is a file containing some JSON:

{
 "exprID":"abc",
 "regex":"[abc]+",
 "description":"ABC sequence finder"
}

we see the following on the terminal:

[Opa] Server dispatch Decoded URL to /expressions
[Opa] Debug Path is [].
[Opa] Debug Body is {some = hello}
[Opa] Debug something failed
[Opa] Server dispatch Decoded URL to /expressions
[Opa] Debug Path is [].
[Opa] Debug Body is {some = {
 "exprID":"abc",
 "regex":"[abc]+",
 "description":"ABC sequence finder"
}

}
[Opa] Debug got object with fields abc, [abc]+, ABC sequence finder
[Opa] Debug record loaded into database...try a get now

Of course nothing got written into the database (line was commented out), but for the first call we can see that we hit the default match clause for an invalid JSON object. In the second we can see the body and the output of the debug statement that accesses the Opa object directly. So it seems to work.

NB: as I stated earlier, I'm not strictly to my specification here, nor am I being too strict about error handling - this is somewhat deliberate for the moment.

FINALLY, the database: We'll uncomment the line:

/regexDB/expressions[e.exprID] <- e; 

which states that in the expressions part of the regexDB which you recall was a stringmap of regexExpressions, we use the exprID field of "e" as the key and use the whole record as the value associated with this key. Simple addition of a record to a hashtable.

A quick note on deployment: Opa integrates with MongoDB. If you have MongoDB installed and running then fine, if not please refer to their documentation on this. If MongoDB is not installed then Opa downloads it - I'm not 100% clear on what it does but I like to try and keep things neatly managed so I know what is going on so I try to avoid this. I will assume therefore that you have MongoDB running on your local machine. Note the parameter to tell Opa that you wish it to use that instance of the database:

ian@U11-VirtualBox:~/opatutorial$ ./tutorial2.js --db-remote 127.0.0.1:27017
/home/ian /home/ian/.opa
http serving on http://U11-VirtualBox:8080
[Opa] regexDB DbGen/Mongo/SynchroStart Opening database
[Opa] MongoDriver.open 127.0.0.1:27017
[Opa] regexDB DbGen/Mongo/SynchroStart Db is ready
[Opa] regexDB DbGen/Mongo/SynchroStart Process 0 operations on the db wait list, start
[Opa] regexDB DbGen/Mongo/SynchroStart Process 0 operations on the db wait list, finished

So now issue the curl command as earlier

ian@U11-VirtualBox:~/opatutorial$ curl -X POST -T "regex1" http://127.0.0.1:8080/expressions

and I promise, somewhere in MongoDB lies a perfectly formed record. Which of course we can't see because we haven't written any GET statement and in all great academic traditions, this is left as an exercise for the reader.

To summarise:

• we created a type
• a database to store records of some type
• extended the pattern matching to process certain paths differently
• met option types
• caught an http method
• deserialised some JSON into Opa
• started Opa with an external database (and even maybe installed and run MongoDB along the way)
• stored a record in that database via a REST call

In the next part I want to clean up the code and make it conform to the specification, write the GET, PUT and DELETE statements, correct the sanity checking and error reporting; and try and validate as much against specification as I can.

Finally, while writing this I was reminded of this quote by Kernighan and Pauger [2]:

 “Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?”

as I'm writing this tutorial and explaining what I'm doing I'm very quickly finding all those places where my code isn't great, my spec isn't great and more importantly forcing myself to really understanding what is going on...that's the only way you're going to get great code at the end of the day...explain it!

References

[1] Finding or Verifying Credit Card Numbers
[2] B. W. Kernighan and P. J. Plauger, The Elements of Programming Style 2nd Edition, McGraw Hill, New York, 1978. ISBN 0-07-034207-5

Opa Language Tutorial: Part 1

I've been experimenting with the Opa programming language for a couple of weeks now and, for me, it addresses various issues of writing web/cloud services really quite well. It has a bunch of features and some very nice abstractions that make the job quite well. In a nutshell it combines the features and semantics of OCaml, Erlang and Javascript and binds them together. I'm not going to draw comparisons with other languages and frameworks - I'll leave that to others, but suffice to say, I like Opa and if it does the job, in this case really quite well, then I'm happy.

So that aside I though it would be nice to present some of my results and experiences of application design and programming in Opa as a tutorial. Opa already has a good overview tutorial and there's further discussion on StackOverflow, the original Opa tutorial (a tour of Opa) and on the Opa forums, but nothing beats learning a language by actually applying it to a real problem and within the process and method structures found in practice.

So what we're going to do is develop a system or application for managing regular expressions. Just something simple enough to easily build, understand and use enough basic features to make an interesting application. Here are the requirements:

1. Store regular expressions with descriptions
2. Store logical sets of regular expressions with descriptions
3. Present the data back in JSON format
4. Have some nice reports and static front-end to this database for documentation, debugging etc...
5. Must be REST
6. Must be "Cloudified"

OK, the last two are architectural in nature and part of the current web/cloud zeitgeist.

The REST paradigm has some good, architectural ideas and fits in quite well with that other love of mine: formal methods. Now before you run away screaming saying that "agile" is the way to go and we don't need that stinking design stuff, agile is just a way a managing your processes such that you focus on what functionality to produce next in order to immediately satisfy the customer. In order to understand and prioritise you need to understand what you're doing and effectively communicate that - formal methods give us that clarity. This isn't a tutorial (nor argument/discussion - I've written about this before here and here) about agile and formal methods but I'll be using their principles (see [1]).

But first, let's get started with some design. REST and web design talks about resources which are access via HTTP verbs and URIs. Skipping a long thought process, let's partially tackle requirement #4 and we get something like:

Resource	GET
/	Return welcome page. Always successful (HTTP 200 OK)

Let's now write some Opa.

I assume you have Opa installed and are comfortable with programming in general. Here's the entire code for this part of the tutorial:

function hello() {
   Resource.styled_page("Expressions Server - Hello", ["/resources/css.css"],
      <div>
         This server contains various regular expressions for data analysis either presented individually or by grouping and returned as JSON objects via the REST interface. See API documents for more information
       </> );
}

function error() {
   Resource.styled_page("Expressions Server - Error", ["/resources/css.css"],
      <div>
          No idea of what you want, go read the API documents for the web and REST/JSON interfaces, or better still go read the source code!
      </> );
}

function start(uri) {
   match (uri) {
       case {path: [] ... }: hello();
       case {~path ...}: error();
   }
}

Server.start(
   Server.http,
     [ {resources: @static_include_directory("resources")} , {dispatch: start} ]
);


The two functions hello() and error() should be self-explanatory above - their syntax isn't too different from Java, C++ etc. Inside however we have a single statement:

Resource.styled_page("Expressions Server - Hello", ["/resources/css.css"],
      <div>

         This server contains various regular expressions for data analysis either presented individually or by grouping and returned as JSON objects via the REST interface. See API documents for more information
       </> );

Resource.styled_page is an Opa function that returns web pages with a title, a attached files and some content.

• The first parameter is a string containing the title of the page, in this case it is the string  "Expressions Server - Hello".
• The second parameter is a list, denoted by square brackets and a comma separated items (we have only one item here). Each item is a string containing a link to some style resource. In this case the link is /resources/css.css which points to some (currently undefined) CSS file. 
• The last entry is XHTML containing the content of the page.

Opa treats XHTML as part of the language and allows Opa code and XHTML to be freely mixed - cool feature! Strictly speaking XHTML in a datatype in Opa recognised by the use of opening and closing tags, in this case we're using div tags. We'll look more at what you can do with Opa and XHTML later but now we can see that this parameter in the above function simply presents us with a div section and some welcoming text.

When called Resource.style_page places all this together and returns a resource object which will be presented back to the caller as, effectively, a web page with DOCTYPE and other necessary structures. More strictly a Resource is anything that can be accessed by a URL, eg: webpage, picture, XML, JSON object etc...

Opa treats the last line of any function as the value to be returned by that function. Some languages use an expilcit return statement whereas Opa doesn't - this belies somewhat Opa's functional heritage.

Let's now jump to the Server.start() function:

Server.start(
   Server.http,
     [ {resources: @static_include_directory("resources")} , {dispatch: start} ]
);

Server.start() is Opa's equivalent of main(). In this case we're passing two parameters:

• the first is the configuration of the server - in this case Server.http is a convenience function that returns a suitable configuration for a basic, default http protocol server.
• the second is a list of handlers and resources for the server. This appears to be a fairly complex and sophisticated structure, here we provide two parameters: a static resource definition and a function that gets calls when the server is accessed.

The interesting thing here is the dispatch function: start(). When we call our http server, it in turn dispatches that call to a function called start() which is where the interpretation of whatever we send to the server is done:

function start(uri) {
   match (uri) {
       case {path: [] ... }: hello();
       case {~path ...}: error();
   }
}

Dispatch functions like start(), hard to believe, return objects of type Resource - we'll come to this little piece of static typing in a moment. For now start() takes a URI as parameter, this is passed by the server. So if I point my browser to "http:/127.0.0.1:8080/xyz" then the URI received by start() is "http:/127.0.0.1:8080/xyz".

The match-case takes this URI and tries to match it against the cases listed. URI's are actually structures containing fields such as path, query fields and port etc. In this case we're interested in the path. The syntax looks a little confusing but match passes the URI structure to each case statement, which then calls the path attribute of that URI. The case statement:

case {path: [] ... }:

tries to match the path of the URI against an empty list pattern. Uri paths are lists of strings if you're wondering. If a match is made then the function hello() is called.

If we pass the URI "http:/127.0.0.1/" to start() then the variable uri then the path of uri (ie: uri.path) will be empty - there is no path - and the empty list is denoted [] in Opa.

When we call the function hello() a Resouce object as described earlier is returned, this in turn is returned from the function start() back to our Server and ultimately to the program which called the server.

The second statement in the case pattern matching matches everything else. The tilde character (~) here states that is the pattern "path=path" matched which happens to be tautology and we call the function error() which behaves similarly to hello().

I guess that more or less explains the code. So let's compile it:

ian@U11-VirtualBox:~/opatutorial$ opa tutorial1.opa
Warning unused
File "tutorial1.opa", line 24, characters 14-18, (24:14-24:18 | 705-709)
  Unused variable path.
Warning inclusions.directory_empty
Directory /home/ian/opatutorial/resources is empty.
ian@U11-VirtualBox:~/opatutorial$

The command to compile is "opa" followed by the name of the file. The extension is optional but I use .opa for identification. You'll notice the compiler complaining about unused variables and empty resource directories. Take all warnings seriously:

• in line 24, case {~path ...}: error(), we just don't use path for anything other than the pattern matching, so this is fine(ish).
• We've included information about resources and their location but not provided anything at runtime. We don't worry about this now, I've no CSS files there anyway.

The compiler generates an executable, which in this case happens to be a file containing JavaScript and requires the existence of node.js on your system. You might need to make this file executable (I'm using Linux for these examples, never tried this on Windows nor Mac sorry) and the file contains the necessary headers to allow the various Linux shells to execute the script from the command line. Note, it may take a short while to start up

ian@U11-VirtualBox:~/opatutorial$ chmod u+rwx tutorial1.js
ian@U11-VirtualBox:~/opatutorial$ ls -l tutorial1.js
-rwx------ 1 ian ian 6171 Jul 28 10:59 tutorial1.js
ian@U11-VirtualBox:~/opatutorial$ ./tutorial1.js
http serving on http://U11-VirtualBox:8080

You can leave out the chmod and ls in future, these are there just for explanatory purposes here. Once the executable reports the http server is running and where - note the default use of port 8080, we can either conenct a browser there or use something like curl:

ian@U11-VirtualBox:~/opatutorial$ curl -i -X GET http://127.0.0.1:8080/

HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Date: Sat, 28 Jul 2012 08:03:16 GMT
Server: http
Content-Type: application/xhtml+xml; charset=utf-8
...

NB: I've cut most of the output just to show the success message (HTTP/1.1 200 OK) from the server. In a browser window it looks like so:

and if we use a longer path (any longer path in this instance will work):

As you can see both of the above correspond to the pattern matching and the calls we make to the hello() and error() functions for this statically defined content in our application.

To close the server use Ctrl+C (or kill -15, kill -9 as necessary)

So that's it for the moment...a fully functional, running web server with a couple of static pages. Not much, but it is a start...next time we'll get in to the database and some REST calls.

References

[1] Ian Oliver (2007) Experiences of Formal Methods in Conventional Software and Systems Design. BCS-FACS Meeting on Formal Methods in Industry. December 2007. London, UK

R-Pi and the mainframe

As far as I can work out I'm probably the second person to do this, and the first in Finland. Andrew Back's blog has an Raspberry Pi running Hercules, and in a later post SimH and VMS.

The Hercules mainframe emulator compiles successfully under Debian on the Raspberry Pi. With a few minor tweakings to get gcc to accept the armv6l parameter to the --mcpu and --march options. Surgery to the config.status file removing every reference to armv6l worked (not a pretty solution!).

Compilation took around 90 minutes with about 10-15 for the make install at the end. Typing "hercules" at the command prompt gives:

NB: Ignore the date, I was too lazy to set the R-Pi's clock..

So hercules compiles and runs. Now just waiting for the download of Volker Brandke's Turnkey MVS 3.8j distribution via Jan Maynard's IBM public domain software collection and in an hour or so should have MVS running on an emulated IBM mainframe on an 32 euro (in P&P) single board computer. Anyone fancy doing the performance calculations between the R-Pi and the original mainframe?

Why you may ask? Well, a credit card sized 1970's mainframe is...well...ok, I have no good answer apart from it's fun...which is what the R-Pi is all about...

Semantic Isolation (Pt. 2½)

This is part 2.5, I'm reserving part 3 for the deeper semantic work and I needed to write some  notes after spending a week working on unifying a set of data-sets for our analytics teams -  extremely interesting and surprisingly challenging (in a good way!). This also has links with privacy and understanding what can and can not be linked and the semantics of those linkages is critical to enabling consumer privacy and compliance.

The unification of identifiers was presented in part 2 (link to part1 here too) as a way of  establishing links between two disparate data sets and breaking the data siloing that occurs information systems.

We start with the premise established earlier: The structure of the identifiers is considered a compound key to the records in that data set and understanding this structure is key to breaking the data siloing (see Apps Considered Harmful).


To give semantics (ostensibly a denotational semantics) to those identifiers we map these into "real world" structures which represent exactly to what those identifier should refer to. One of the discoveries here has been that it has been assumed that, for example, a person identifier always refers to a real- world person, or that a device identifier (eg: IMEI, IMSI) refers to an actual device and that there is a one-to-one correspondance between devices and people.

Note: this isn't necessarily a good model of the real-world, the question is that does this model suffice for the context and purpose to which it is being applied.

However common identifiers such as user IDs, IMEI, IMSI do not refer to persons and devices directly but often through artifacts such as SIM cards and a person's persona. Adding to this complexity is that the users and owners of devices change over time, and that we now have mobile devices which support multiple SIM cards. At any point in time we might construct a model of the real-world artifacts thus:

Typically analyics over these data sets - which is the driver for unification to enable  information consistency and quality over cross-referencing of multiple data sets - takes a  particular period in time, say 1 day to 3 months, so that we can dispense with dealing with  certain changes. The longer the analytical period the lower the data quality and there's an  interesting set of research that can be made there on measuring the quality loss.

So the main findings so far are:

• It is given that each user identifier (user ID, email address) used for uniquely identifying  users is assumed to be a separate person.
• It is generally assumed that equipment identifiers and addresses (IMEI, IMSI, IP address)  identify unique pieces of equipment
• The relationship between a device and a person/user is 1-to-1

We have no notion of persona, in fact I've never seen any system with a good notion of  persona. Given two identifiers such as email address as used as username, then two email  addresses used by the same person are assumed to represent two persons. The typical use for  this is to allow a user to differentiate between two uses such as one for social purposes and  one for business purposes. The complicates of linking arises because of the strongly  directional nature of the person to persona relationship - in the discrete terms of UML and ER  modelling, this is a simple directional relationship of the form:

Aside: the notion of persona is best explained by Jung as 'a kind of mask, designed on the one  hand to make a definite impression upon others, and on the other to conceal the true nature of  the individual' which is found in Two Essays on Analytical Psychology. Probably we take a much  more crude view on persona but the principle is the same. There are other views of this such as those explained here: True self and false self.

Just to complicate the above we probably can not assume that a set of personas always refer to  the same person.

The situation with device identifiers and devices is remarkably similar and this is really being highlighted by the emergence of factors such as multiple-SIM, multiple device ownership and misguided collection of identifiers resulting in complicated and error-prone cross-referencing between the plurality of these from data sets of varying quality and information content.

Aside: we haven't dealt with information hiding through encryption and hashing yet and here we either rely upon knowing decryption keys, pattern or semantic matching or weak hashes and trivial salting.

For the moment we have the conclusion that the mapping between identifiers in data sets often do not match expectations of what it is assumed they actually identify and that assumed relationships between real-world artifacts are made implicitly without respect for the actual usage and meaning of those artifacts.

Just to finish (this might become part 2.75) that the data transformation processes:  filtering, abstraction etc over a data-set or extracted portion thereof often implicitly  refines the relationships in the identifier structures themselves and also at the semantics or  real-world level.

More on "Apps considered harmful"

Just a heads up that the abstract to Ora's keynote at CIDOC 2012 in Helsinki in up on the CIDOC pages at the following address: http://www.cidoc2012.fi/en/File/1575/lassila-abstract.pdf

It is entitled: Love Thy Data (or: Apps Considered Harmful) and I've seen the slide-set...something really worth waiting for and when it becomes publicly available I'll post the link here as well.

No more interest in The Cloud?

Forbes has an article on the level of interest in cloud computing stating that the concept has moved, according to the Gartner Hype Curve into the so called 'trough of disillusionment'.

"Lately I’ve been hearing some rumblings during my various discussions around cloud computing. Some in the industry have been quietly saying the end is near for the much over hyped term.  I wouldn’t go as far as to say the cloud is dead just yet, but there is a growing sense that  interest in cloud computing, at least from the point of view of a buzz word, has peaked." 

Reuven Cohen, Contributor, Forbes, 24 May 2012

Now something has worried me a little in how "cloud" has been approached is that it appears in most cases to be overwhelmingly oriented towards either data storage or as a mechanism for off-loading or virtualising IT infrastructure - in the latter case, not running your own computers but just running your computers even though they are virtual machines.

At least from the desktop or home user, cloud is little more than data storage, eg: Flickr, Youtube, and at best a simple sharing mechanism.

My personal feeling here is that we're really missing the killer application (not the killer 'app') that would truely start to liberate computing from being locked to a single computing element (eg: my laptop) or some super client-server thing into a totally virtualised computing entity with processes and data (or even information!) being truly mobile according to computation need.

One could argue that Google is currently providing something close to this, but I still think that the granularity and information integration needs to decrease and increase respectively to approach what I'm looking for.

Maybe there isn't a killer application...for example, the way the Semantic Web could become the killer platform on top of the cloud? In this sense then cloud becomes the killer enabler.

Either way, 'cloud' was always going to be a disappointment; or am I juts being cynical under a barrage of buzzwords?

Apps considered harmful

OK, first of all this is joint work with my great colleague Ora Lassila. I've just come off the phone with him discussing our World Domination Plan(TM), or how to unleash the power of the Semantic Web and revolutionize computing...anyway, Ora is in Helsinki next month giving a keynote speech at a conference on museums and we were discussing the title based on various thoughts of ours regarding "apps" and previous work of ours on cloud-based, semantic information systems, eg: sTuples [1] [and Sedvice/M3 [2] (also see here).

One of the things we vehemently agreed upon was that "apps must die" for the reason that "apps" are a new way of isolating (both physically and semantically!) data and preventing interactivity; whereas Sedvice/M3 specifically was designed to break down the semantic isolation and ultimately reduce "apps" to be nothing more than domain specific user-interface manifestations of information - and ideally in some cases, become a novel user-interaction mechanism.

After tonight's discussion, I make a quite search on the "apps considered harmful" meme and found an article "Apps Considered Harmful: Part1" at Elderlore's blog. I really like this quote:

"So,what can we do? In part two of this article,I will talk about moving beyond the fragmented experiences of App World into something more interesting,where data and programs are explicitly represented as objects that you are empowered and invited to interconnect. Consider something like GNU Emacs:the separate “applications”in the Emacs Lisp space are delivered as bundles of functions and variables and hooks,not as single-screen apps that cannot interact or share data. You get things done in Emacs by customizing variables and hooking things together;potentially,any function in any Elisp application can call any other function or access any other data in the system,regardless of which package implements the functionality,and most of the coolness of emacs is in being able to coordinate things this way when solving problems,and to save these coordinations as resuable elisp code."

and also

"I want to smash the App World into pieces;that is to break apps into smaller,reusable pieces or “blocks”which can be visually clicked together like MIT Scratch,but on an industrial scale. Apple already banned this type of software from the iPad;perhaps they know something they aren’t telling us?"

So do we! So de we! Even the EU has started talking about data liberation!

Much of the reasoning behind promoting the "Apple style of app" (remember Nokia's Widsets made years earlier - actually they were coming quite close to our ideal?!) is based around locking-in the developer and especially the user. Indeed pretty much every piece of functionality in most "apps" found on all smart phone operating systems is for consumption of data and recording in specific detail the user's behaviour associated with that particular consumption interaction. Ironically all that functionality is present in the generic browser (eg: Opera, Firefox etc) and usually, much more control over your individual privacy too.

Privacy however is a side issue here for the moment and our focus is actually on what an "app" is in the information consumption context above. Indeed as we demonstrated in M3 it is little more than a query over an information space. In the case of M3 and the RDF/OWL graph view of information, a navigation between chunks of information. Indeed this is the basis of Facebook's social graph.

The power here doesn't just come from the ability to easily navigate over information but also the ease in which reasoning is built on top; from the simple subclassing inferences to complex context specific, non-monotonic rules.

Through this reasoning and suitable ontological structures it becomes almost trivial to add functionality into the graph, eg: new messaging types, and apps which are designed to query for super-types of those automatically pick-up on these new objects. This is hard to do in an "app" - which are more like the traditional applications of old: fixed and unexpandable - who thought of Twitter and Facebook contacts when writing contact book apps all those years ago?

The point where we're heading here is that what we think that "apps" should have been are just queries over information with reasoning at the information store level, that is, pushing the logicl that gives semantics, interoperability and expandability down into the lower-levels of the information stack.

One can argue that "apps" in their current manifestations are optimised for the devices they run upon, eg: an app for iOS delivers the content in the form best for that device. However we would go further (and we made an amazing demo with M3 of this back in 2009), where we encoded how the information should be displayed in various manifestations, eg: list, tile etc - there was an ontology for this, and then let the device pick the best rendering for that informtaion based upon its local context. What made this interesting was the reasoning mechanism searched the information itself, the contained or linked chunks of information as well as searching the type hierarchy for display hints.

We did the same with actions too...and we even started on serialising computations and passing these around...true cloud computing...quite away from the humble, locked-in current definition of "app"...

References

[1]Deepali Khushraj, Ora Lassila, Tim Finin. (2004) sTuples : Semantic Tuple Spaces. Access (2004) IEEE, Pages: 268-277

[2] Ian Oliver, Jukka Honkola (2008) Personal Semantic Web Through A Space Based Computing Environment. Middleware for the Semantic Web Second IEEE International Conference on Semantic Computing.