An article on The Semantics of PII

A while back I wrote a short article for the IAPP's Privacy Tech Blog. With permission I'll reproduce it here for additional reference. Also,a tip of the hat for the administrator of the blog: Jedidah Bracy of the IAPP for his spell checking, grammar checking and editorial skills!

The Semantics of PII

Privacy Tech | Feb 26, 2015

Last year, Profs. Peter Swire and Annie Antón wrote a compelling piece in Privacy Perspectives about the need for privacy engineers and lawyers to get along. Establishing a common language in which to communicate will be essential to appropriately connect policy with technology.

It’s probably safe to say that the most common terms used in privacy are personally identifiable information (PII) and personal data, depending upon whether you come from a U.S. or European background. I think these terms are more or less self-explanatory.

But what do they really mean?

Take PII, for example. It means a chunk of data that reveals some knowledge about a person that can be unambiguously identified. Sounds more or less about right, doesn’t it? Is a computer's IP address personally identifiable? What if that IP address belongs to a router for a large, multinational corporation? Is it PII then? And what if it belongs to a family using multiple computers, tablets, phones or other devices?

We will soon find ourselves delving into the minutiae of meaning—the what-does-personal-really-mean type questions. Plus, we must ask what isinformation, and what does identifiable denote?

There is a whole area of linguistics, philosophy and mathematics—take your pick—that deals with the meaning of things, otherwise known as semantics, or even semiotics if you want the overall field.

Mathematicians took years to fully understand the semantics of even simple statements such as 1+1=2, which looks obvious until you try to explain what 1 is, what 2 is, what + means, what = means and then what it means to say 1+1. The English philosophers Bertrand Russell and Albert Whitehead spent most of their careers writing Principia Mathematica to answer this question, and after four editions and 300 pages of dense mathematics, they had an answer. That was, until a young German by the name of Kurt Gödel came along and shook mathematics to its foundations with an equally "trivial" result.

So if it took 300 pages by two of the brightest minds in mathematics to give us a semantics for 1+1=2, how many pages—and years of work—will it take to give "PII" a semantics?

Now here's an interesting point: The definition of PII that is used in contemporary privacy is perfectly well defined in the privacy-legal context. I can go to various legal documents and read a formal definition of what PII or personal data means. But as we move between disciplines—in our case from privacy-legal to privacy-engineering disciplines—these definitions no longer hold, or at the very least, they don't work well.

If we move to the other end of the scale from legal to mathematics, we find concepts such as information entropy, which provides a clear, unambiguous and precise definition of what information is as well as the identifiability of a data set with respect to some population and so on. Information entropy, however, is not an easy concept with which to work. We can state now that the legal definition of PII can be defined in terms of the mathematical definition; it's just that this is obscenely difficult to do.

Somewhere between these two extremes lies software engineering, the discipline that actually implements privacy law into our systems, in ostensibly mathematical (programming language) terms.

Software engineers, much to the chagrin of privacy lawyers, do not understand legal terms. Well, ok, they do to a point, but you try coding a statement such as "reasonable privacy" into C++ or Java!

Plus, privacy lawyers don't understand all the subtle ramifications of virtual machines, machine language, object orientation, distributed computing, network protocols, XML, RDF—the list goes on!—again, much to the chagrin of software engineers.

Yet, as we stated earlier, there is a relationship between the terms and language that privacy lawyers use and the terms and language that software engineers use. That link provides the translation mechanism that allows both groups not just to talk but to properly communicate with each other.

We can spend as much time as we’d like writing manifestos and principles, designing processes, inventing new job titles such as privacy officer, privacy compliance tsar, grand chief-overseer-of-the-worshipful-court-of-privacy-dudes and so on, but without grounding semantics into terms such as PII and personal data—terms that will allow us to translate between legal-speak and engineer-speak—all of this work will be in vain.

First International Workshop on Privacy Engineering (IWPE'15)

CFP:

First International Workshop on Privacy Engineering

(IWPE'15)

http://ieee-security.org/TC/SPW2015/IWPE/

http://www.ieee-security.org/TC/SP2015

21 May 2015 - The Fairmont, San Jose, CA 

---

IMPORTANT DATES:
Deadline of paper submission:  23 January, 2015
Notification of acceptance:    16 February, 2015 
Accepted Paper camera ready:   3 March, 2015  

---

We are pleased to invite you to participate in the premier annual event of the International Workshop on Privacy Engineering (IWPE'15).

Ongoing news reports regarding global surveillance programs, massive personal data breaches in corporate databases, and notorious examples of personal tragedies due to privacy violations have intensified societal demands for privacy-friendly systems. In response, current legislative and standardization processes worldwide aim to strengthen individual’s privacy by introducing legal and organizational frameworks that personal data collectors and processors must follow.

However, in practice, these initiatives alone are not enough to guarantee that organizations and software developers will be able to identify and adopt appropriate privacy engineering techniques in their daily practices. Even if so, it is difficult to systematically evaluate whether the systems they develop using such techniques comply with legal frameworks, provide necessary technical assurances, and fulfill users’ privacy requirements. It is evident that research is needed in developing techniques that can aid the translation of legal and normative concepts, as well as user expectations into systems requirements. Furthermore, methods that can support organizations and engineers in developing (socio-)technical systems that address these requirements is of increasing value to respond to the existing societal challenges associated with privacy.

In this context, privacy engineering research is emerging as an important topic. Engineers are increasingly expected to build and maintain privacy-preserving and data-protection compliant systems in different ICT domains such as health, energy, transportation, social computing, law enforcement, public services; based on different infrastructures such as cloud, grid, or mobile computing and architectures. While there is a consensus on the benefits of an engineering approach to privacy, concrete proposals for processes, models, methodologies, techniques and tools that support engineers and organizations in this endeavor are few and in need of immediate attention.

To cover this gap, the topics of the International Workshop on Privacy Engineering (IWPE'15) focus on all the aspects surrounding privacy engineering, ranging from its theoretical foundations, engineering approaches, and support infrastructures, to its practical application in projects of different scale. Specifically, we are seeking the following kinds of papers: (1) technical solution papers that illustrate a novel formalism, method or other research finding with preliminary evaluation; (2) experience and practice papers that describe a case study, challenge or lessons learned from in a specific domain; (3) early evaluations of tools and other infrastructure that support engineering tasks in privacy requirements, design, implementation, testing, etc.; (4) interdisciplinary studies or critical reviews of existing privacy engineering concepts, methods and frameworks; or (5) vision papers that take a clear position informed by evidence based on a thorough literature review.

IWPE’15 welcomes papers that focus on novel solutions on the recent developments in the general area of privacy engineering. Topics of interests include, but are not limited to:

• Integration of law and policy compliance into the development process
• Privacy impact assessment
• Privacy risk management models
• Privacy breach recovery Methods
• Technical standards, heuristics and best practices for privacy engineering
• Privacy engineering in technical standards
• Privacy requirements elicitation and analysis methods
• User privacy and data protection requirements
• Management of privacy requirements with other system requirements
• Privacy requirements operationalization
• Privacy engineering strategies and design patterns
• Privacy architectures
• Privacy engineering and databases
• Privacy engineering in the context of interaction design and usability
• Privacy testing and evaluation methods
• Validation and verification of privacy requirements
• Engineering Privacy Enhancing Technologies
• Models and approaches for the verification of privacy properties
• Tools supporting privacy engineering
• Teaching and training privacy engineering
• Adaptations of privacy engineering into specific software development processes
• Pilots and real-world applications
• Privacy engineering and accountability
• Organizational, legal, political and economic aspects of privacy engineering

This topic list is not meant to be exhaustive; since IWPE'15 is interested in all aspects of privacy engineering. However, papers without a clear application to privacy engineering will be considered out of scope and may be rejected without full review.

---

PAPER FORMAT & SUBMISSION

We solicit unpublished short position papers (up to 4 pages) and long papers reporting technical, research or industry experience (up to 8 pages) on all dimensions of the privacy engineering domain. Each paper, written in English, must follow IEEE Proceedings format. Submission of a paper should be regarded as an undertaking that, should the paper be accepted, at least one of the authors will attend the workshop to present the paper. All papers must be submitted via EasyChair at https://www.easychair.org/conferences/?conf=iwpe15

All IWPE'15 Papers will be published in IEEE eXplore, which is indexed by EI Engineering Index, ISI Conference Proceedings Citation Index (CPCI-S), Scopus etc.

---

If you have any questions regarding IWPE'15, please contact:

Jose M. del Alamo (jm.delalamo@upm.es)
Norman Sadeh (sadeh@cs.cmu.edu)
Seda Gurses (seda@nyu.edu)
Dawn Jutla (dawn.jutla@smu.ca)

Wither the Privacy Hero...

The Hero in computing is a well known phenomena - think of the lone programmer, sysadmin or hacker for example. However the hero also occurs in all domains including privacy. The privacy hero is the one who has hand crafted the privacy policy, set down in stone a lengthly list of privacy requirements and compliance activies without consulting the engineers and users who have to implement and use these.

As many disciplines, especially that of medicine, have discovered, the hero is the most dangerous person there is. By working against the odds, he (or she) usually creates a victory where all is solved [1]. Be it uring a patient or creating the rules by which the company is saved from an inglorious admission of a data breach. Even if there is a breach or the patient later dies, it can't be the hero's fault, but the others such as the failed care of the nurses or the engineers who never listened. In reality the nurses and the engineers are usually patching the damages caused by the hero.

Our current cultural setups in privacy, and especially now we're starting to get engineers actively involved in the privacy debate, needs to change from the Privacy Heros to a much tightly integrated team of experts.

In [2], Atul Gawande clearly states that the nurses, technicians and other personel "work for" the hero doctor. In privacy we still have the same attitude, software engineers "work for" the Privacy Officers.

We suffer from a huge lack of teamwork - the privacy hero's word is the Truth and that's it. Within the current culture of privacy, the engineers who are battling to implement or even comprehend privacy requirements written and explained at a completely different level of asbtraction than is necessary do not play any major part in those requirements.

Consider the defintions of personal data or PII for example, have these even been properly grounded in the undelying mathematical theories of what information is; or even for that matter in terms that can be properly understood by software engineers in their domain. Even within the legal domain, these terms have been defined in such a way that they are underspecified and open to legal interpretation.

In order to move from a highly ineffective privacy priesthood to a true, all encompasing and all relevant discipline based on a mutually supporting combination of legal, scientific and engineering principles we must change our culture from that of the Hero to that of the Team.

References

[1] Suzanne Gordon, Patrick Mendenhall and Bonnie Blair O'Connor. Beyond the Checklist
[2] Atul Gawande, Better
[3] Ian Oliver. Privacy Engineering: A Dataflow and Ontological Approach.

Surgical privacy: Information Handling in an Infectious Environment

What has privacy engineering, data flow modelling and analysis got to do with how infectious materials and the sterile field are handled in medical situations?  Are there things we can learn by exploiting by drawing an analogy between these seemingly different fields?

Surgical privacy: Information Handling in an Infectious Environment from Ian Oliver

We've discussed this subject earlier and a few links can be found here. Indeed privacy engineering has a lot to learn from analogous environments such as aviation, medicine, anaesthesia, chemical engineering and so on; the commonality here is that those environments understood they had to take a whole systems approach rather than relying upon a top-down driven approach or relying upon embedding the semantics of the area in one selected discipline.

Privacy and Big Data in Medicine

A short article by myself on the subject of privacy in medicine was just published in the web magazine Britain's Nurses. Quite an experience writing for a very different audience than software engineers, but extremely interesting to note the similarities between the domains.

When it comes to privacy, one of the seemingly infinite problems we face is how to develop the techniques, tools and technologies in our respective domains. Here again we have the choice of reinventing the wheel or looking to different domains and use their knowledge and experiences. This latter route is the much preferred but rarely taken.

So for the moment, I'll take the chance to look back on previous articles that draw lessons from other domains:

• Privacy learning from aviation safety...
• What has surgery got to do with information privacy?
• Understanding Software Engineering Accidents
• Flying planes, Surgery and Privacy - the presentation
• Systems Safety - Defining Moments

Domains such as medicine, civil engineering and especially aviation have been through this process and as information rises in value - that is the economic effects of a data breach or loss of consumer confidence - reach levels where companies will figuratively crash, so the need to take in these learnings and treat information handling as any other element in a safety-critical system.

Finally the article I mentioned: Privacy in Digital Health, 12 May 2014, Britain's Nurses

Airbus Aircraft Operational Philosophy

I'm reading the Airbus Flight Crew Training Manual for the A320/A321 aircraft at the moment. If anything, to get an understanding of how such things are written and presented. Aircraft are fairly complex systems and as I've already talked about in this blog, many ideas are directly applicable to privacy, security, software engineering etc if we only take time to learn.

Aside: If anyone does own an A320 or equivalent simulator and would like to offer me some time flying (simulated or otherwise) then YES PLEASE!!!.  OK, back to reality now...

A few things struck me as being particularly relevant such as The Operational Golden Rules:

1. The aircraft can be flown like any other aircraft
2. Fly, navigate, communicate - in that order
3. One head up at all times
4. Cross check the accuracy of the FMS
5. Know your FMA at all times 
6. When things don’t go as expected - take over 
7. Use the proper level of automation for the task
8. Practice task sharing and back-up each other

Nothing too surprising there, other than they're stating the obvious.

That's the great thing about the obvious, completely missable...

Note the emphasis on getting the job done without requiring anything other than basic aviation skills; or in our case basic software engineering skills. The emphasis on cross-checking, task sharing and delegation, the use of appropriate techniques and technologies and the note that when things don't go as expected - take over!

As an idea to start off...

1. There is nothing special about this system under development
2. Plan, code, test - in that order
3. Be aware of the wider engineering and requirements context
4. Cross check your system with its original goals
5. ...

The learnings here to our privacy engineering world are deep and profound. Just as an exercise to the reader, look at the procedures being followed for any engineering task and could these be placed into a simple set of operational golden rules as above? There's an interesting comparison waiting to be drawn here...that will have to wait for the moment.

Analysing Data Flow Models

In the previous post we introduced a pattern for the data-flows in and out of an application such as those found on your mobile phone or tablet. In this posting I want to expand on this pattern and explore various annotations to help us reason about how our application treats the information flowing through it.

Let's first introduce an example application, a photo sharing application. This app allows you to select a photo on your device and upload it to a social media account. Here's what it looks like to the user on a mobile device with rounded corners (as opposed to square corners which none of them seem to have):

It looks innocent enough but as we know there are many threats to user privacy even in the most innocent looking of places. So let's model what is really happening behind the scenes. We understand so far a number of things: the user supplies content and credentials for the services, these are stored locally for convenience, the app adds meta-data to the picture before uploading and sends information about the app's behaviour to the inventor of the app. We might then construct the following model:

On each of the dataflows we have noted the kind of information transported over those channels and the mechanism of communication. We also note our presumed trust boundary.

What this shows clearly is where data is flowing and by what means. We have for now skipped over the specific or precise meanings of some things hoping that the terms we have used are self-explanatory.

But, now we have formally written we can focus the discussion on specific aspects of the application, for example:

• What mechanisms are being used to store the user ID and password in the "operating system"? Is this storage secure and sandboxed? I.e. how do we increase the area of trust boundary?
• Are the communication mechanisms from the app to the social media and inventor appropriate?
• What infrastructure information is implicitly included over these channels, for example, IP addresses, user agent strings [1] etc?
• Does the app have authorisation to the various channels?
• What is the granularity of the Location data over the various channels?
• What information and channels are considered primary and which secondary?
• Is the information flowing to the inventor appropriate and what is the content?
• What about the EXIF data embedded in the picture?

Generating a formal list of questions from the above is relatively easy and this is exactly how we proceed.

The next questions which follow are related to how we can reduce the amount of information without compromising the application functionality and business model? For example:

• can we reduce the granularity of the Location sent to the social media systems to, say, city level or country level?
• can we automatically remove EXIF data?
• do we allow the app to work if the operating system's location service is switched off or the user decides not to use this?

And so on...Finally  we get down to the consent related questions

• What does the user see upon first-time usage of the app? What do they have to agree to?
• Do we tell the user what underlying services such as GPS we're using as part of the application
• Secondary data collection opt-out
• For what reason is the data being collected over both primary and secondary channels

And so on again.

What we have done is set the scene, or, circumscribed what we need to investigate and decide upon. Indeed at some level we even have the means to measure the information content of the application and even the extent the consents by implication; and if we can measure then we have a formal mechanism to decide whether one design is "better" than another in terms of privacy.

In the following articles I'll discuss more about the classification mechanisms (information, security, usage, purpose, provenance) and other annotations along with the detailed implications of these.

Notes

[1] User agent strings are very interesting...just ask Panopticlick.

What I look for in a software engineer

When called to assess or interview someone for a software engineering position there are a few things I look for: the core set of knowledge on which we can build upon to support the more specialist or esoteric skills such as security, privacy and the rest. I discussed ideas for what would be required for a grounding in concepts for privacy earlier.

Programming Skill

• This is obviously the first question - "can you program?" and then in what languages. The usual candidates will always be there, ie: Java, Python, C, C++ etc, but I'd like to see things such as SQL, Prolog, Lisp and maybe things like Haskell, ML, Ada, Eiffel etc. Depending upon the position PHP, Perl, Unix shell script etc.
• Ideally the candidate should understand and appreciate the differences and respective advantages of the various programming paradigms: structured, object-oriented, functional etc
• "Programming" in markup languages is not acceptable: HTML IS NOT A PROGRAMMING LANGUAGE.

Design Skill

• The ability to use design languages such as UML, SDL etc, must always be a prerequisite. Correct use of these come with and comprehension of abstraction and refinement of designs.
• Abstraction is a very underappreciated and rarely seen skill. The ability to concentrate on what is required without mixing abstraction levels unnecessarily is one of the marks of an engineer.

Discipline and Formality

• Despite agile being the saviour of everything a true appreciation of what agile actually is, is necessary. The understanding that designs follow a natural flow from concept through design to implementation is invariant - good software can not be hacked.
• Communication is obviously part of this and clear coding, design and concepts is fundamental.
• Experience with formal methods is always a clear advantages. Now whether someone actually uses Alloy, B, Z, VDM etc in another matter altogether; the skills and discipline that comes with these is what we're looking for. Some of the best "coders and architects I have come across are formal methods people at heart; indeed I know of on agile guru who attributes his success to applying formal methods within an agile framework (see communication, abstraction and refinement above).
• This also extends to areas such as testing of code, simulation of design and defensive programming though ideas such as design by contract - all of which contribute to a better quality of development and code.

Fundamentals

• Does the software engineer have a grounding in computer science and/or mathematics? Having this allows the engineer to concentrate on the underlying principles of things without being distracted by arguments over whether something should be represented in XML vs RDF vs JSON, or SQL vs NoSQL etc.

Obviously this just touches the surface of what is required but I hope it gives some idea of what is required. Not all skills are needed, so if someone doesn't have 3 years experience of category theory and programming with monads (the minimum to write "hello world" in Haskell I'm told), but an appreciation and understanding is required.

Ultimately engineering is about solving problem with discipline, rigour and use of the appropriate techniques.

Systems Safety - Defining Moments

As I've been concentrating on "safety improvements", or at least techniques for the improvement of system I've tended to concentrate on four areas:

• Aviation
• Industrial
• Medical (specifically surgical)
• Software Engineering (specifically information privacy)

The parallels between theses areas when it comes to what each define as "safety" and the techniques should be obvious. However the question remains what actually triggered each of these respective areas to take a more inherent safety approach and then what will trigger a similar approach with regards to information safety?

 

Above Diagram Key: Y-Axis: relative degree of safety embedded into that discipline, X-Axis, year of time since seminal incident.

Aviation safety's seminal moment was the 1935 crash of a Boeing Model 299 aircraft during a presentation flight. Instead of blaming the pilots, effort was made to understand the causes of the accident and develop techniques to help prevent similar accidents in the future.

For industrial safety the seminal moment was the 1974 Flixborough Disaster in the UK. This resulted in work on the design of industrial plants and the development of the notion of "inherent safety".

Surgical safety has quite a long tradition especially with the development of anaesthetic safety from the 1960s and the introduction of a proper systems approach. However anesthetists seem not to feature prominently as surgeons and doctors so the fame would probably go to Peter Pronovost et.al. for the Central Line checklist. This was probably one of the major contributors to the WHO Surgical Safety Checklist discussed in detail in Atul Gawande's book The Checklist Manifesto which brings together much of the above incidents. 

If you're still in doubt maybe Atul Gawande's article in the New Yorker magazine entitled The Checklist: If something so simple can transform intensive care, what else can it do? (Dec 10, 2007) might help.

Getting back to the crux of this article, what is the incident that will cause the wholesale change in attitudes and techniques to software engineering that instills such a sense of discipline that we can eradicate errors to such a degree that we could compare ourselves favourably with other disciplines?

The increasingly frequent hacking and information leaks? The NSA wiretapping and mass surveillance? Facebook and Google's privacy policies? None of these have had any lasting effect upon the very core of software engineering if any at all. Which either means that we place such low value on the safety of our information or that the economics of software are so badly formulated in society that the catastrophe would have to be so huge that it would have to cause societal change?

Interestingly, in software engineering and computer science we're certainly not short on techniques for improving the quality and reliability of the systems we're developing: formal methods (eg: Alloy, B, Z, VDM etc), proof, simulation, testing, modelling (in general). What we probably lack is the simplicity of a checklist to guide us through the morass of problems we encounter. In this last respect, this is why I think we're more like surgeons that modern day aviators; or, maybe some of us are like the investigators to the 1935 Boeing crash and other aviation heroes learning their trade?

A basic information privacy programme

Using the material discussed earlier as our starting point we can construct the outline of an information privacy programme (IPP). First however we must place any IPP into juxtaposition with the units it will affect:

Note that any "IPP" can not rule nor dictate over any of the affected parties, it does however provide a point of contact and expertise for those parties when working with information. The enforcement aspects come from a number of other places which we'll discuss in another article.

We structure the IPP into two parts: the programme elements and the knowledge base, both of which we will introduce here.

The Programme Elements are:

• A Reporting System
• An Information Distribution System
• A Privacy Committee
• An Inspection Programme
• Education and Training
• Accident Preparation and Investigation

The Reporting System is a mechanism by which people both inside and outside can bring to attention issues, problems, notification of data breaches and other privacy problems to attention. This mechanism must allow anonymous contributions and can be compared to the aviation related programmes CHIRP and CHIRP-MEMS.

The Information Distribution System is a mechanism by which privacy related topics can be communicated to interested parties outside of the IPP. Typically such information might be of the "interest" variety, eg: notification of new laws or relevant news articles, to the critical such as a notification of an information breach or threat which must be acted upon by any receiving parties.

The Privacy Committee is the one fraught with political difficulties, but a small (5-7 people) group from a cross-section of roles within the company. This is where many programmes fail because of placing management or non-technical staff here. The role of this group is not to manage the programme but to reivew incident/accident reports, review adequacy of corrective actions, hazard reviews, inspection reports, provide recommendations (based on information received via the reportins and information distribution systems) and to self-assess and improve the ways of working of the IPP as a whole. The committee is also the chance to bring onboard the expertise in the external parties from R&D, legal etc.

The Inspection Programme can cover many aspects, but is primarily concerned with auditing and providing mentoring/consultancy to external parties. Auditing usually takes place as part of the deployment cycle of a product for example. This area constitutes most likely the bulk of the work of the IPP.

Education and Training is always critical but often badly done (if at all). Developing relevant training for the external parties is critical to long-term success and accident avoidance.

Finally we get to Accident Preparation and Investigation. Hopefully data breach or similar incidents never happen, however having procedures to deal with these are critical to a good solution to any problem. Likewise, a good investigation procedure which not only traces down the causes but provides solutions is also critical.

This concludes the programme elements and now we move to the Knowledge Base which is where the tools, techniques, metrics and database for storing all of the required knowledge and wisdom for running the above element succesfully. We use the word "tool" in a loose sense - automation in any form might not be available, but things such as checklists or ontological structures/classifications are tools.

As a recommended minimum for databases: a database of relevant laws and implications (at an R&D level of detail), a database of reported risks and a database of investigation findings and so on.

All this is just an overview - specific companies, specific situations require customisation and we still have not discussed in any detail the content of the knowledge base nor programme elements let alone presented particular roles such as that of the privacy officer etc. What however should be clear is that a structuring is necessary and what have presented here summaries some of the best practices from an area that is already well versed with similar issues, that of aviation safety. [1]

References

[1] Wood R. (1997) Aviation Safety Programs. 2nd edn. Jeppesen.

Privacy learning from aviation safety...

I've written before on my thoughts, or belief, that information should be treated as a dangerous chemical, or a patient undergoing surgery:

privacy is a safety critical concern

Developing ideas such as checklists from aviation, surgery etc, do improve the quality of work and actually implementing those has been a very interesting experience - if anyone wants to learn about this let me know, I'm very happy to talk about it.

Now while we have some parts of the "coalface work" working, ie: the real privacy engineering work with real software engineers, software architects, programmers and code, we can now turn to reassess how we're managing privacy and in turn managing risk and improving the safety of our code, products, business and ultimately our customers from a privacy perspective.

One area that has succeeded very well is aviation - our original starting point for much of this safety discussion. Indeed aviation safety precedes most other safety programmes by decades. I'd like then to quote from the book Aviation Safety Programs (Richard Wood, 2nd edn 1997, Jeppesen):

Safety is not a moral problem or an ethical problem or a pain and suffering problem. It is an economic problem. Safety, by itself, generates a lot of sympathy, but very little action. Only the economics of safety generate action. In the entire history of safety, nothing good happened unless it was either economically beneficial or mandated by government.

Now simply replace "safety" by "privacy":

Privacy is not a moral problem or an ethical problem or a pain and suffering problem. It is an economic problem. Privacy, by itself, generates a lot of sympathy, but very little action. Only the economics of privacy generate action. In the entire history of privacy, nothing good happened unless it was either economically beneficial or mandated by government.

Interesting don't you think?  Ah, yes, we know it is an economic problem and it certainly is being mandated by government. Adopting something like Privacy by Design today will not change the quality of your product or relationship with your customers. You'll get some sympathy but ultimately does anything change? There's an outcry every time Facebook changes it privacy policies but people still post private information there; similarly with Google and the rest. Privacy is talked about in moral and ethical terms: building trust with the customer.

Despite of everything, Facebook hasn't lost 50%, 90% or even 1% of its customer base because of its perceived "anti-privacy" stance, and neither has Google, nor any other company I can think of. Privacy must be framed as an economic argument: if company X abuses their consumer data privacy, then they lose all their customers and business.

We've a lot to learn from a successful safety program and that's something I'm going to write about more in the coming days and weeks. How do we put a privacy programme together such that it really does economically benefit a company and addresses all the cross-cutting issues from the top management through to the programmer actually writing the code that runs the business?

First things first though, when constructing such a programme, we need a group of people to lead the privacy culture within a company and that team needs day-to-day experience close to coalface, they need knowledge of not just the moral and ethical aspects but of real scientific and engineering experience. Starting with that we might just really see where the economics benefits are won and lost.

On The Naivety of Privacy

Recent events regarding privacy and the internet have left me wondering if we are being somewhat naïve. We are starting to see a slew of new laws, strategies and technologies for protecting our privacy in what is effectively a public space. The end-user however is not, as far as I can tell, really getting the benefit of this - indeed if anyone is it is the emerging privacy-industrial complex [1] as some have written.

It is utterly naïve to believe that laws, strategies, intentions, grand speeches, certifications, automatic filtering, classification iconography and so on make for better end-user privacy. The more we do this the more confused we become, and simultaneously we lose sight of what we're really trying to achieve. Spare a thought for the poor end-users.

There is a great deal that is misunderstood or not known by privacy advocates about how the internet, computers and information systems work - I fear in a lot of cases either some don't want to understand because it takes them outside of their comfort zone, or the semantic gap between the engineers and the legal/advocacy side is too great and that bridging this gap is extraordinarily difficult for both parties.

I worry about our lack of formality and discipline, possibly in equal quantities. We – the privacy community – lack these aspects to really understand and accept the fundamentals of our area and how to apply this to the information systems we are trying to protect. In some cases we are actively fighting against the need to scientifically ground our chosen area.

We must take a moment to think and understand what problem we are really trying to solve. The more philosophical works by Solove and Nissenbaum address the overall concept of privacy. I'm not sure that the implications of these are really understood. Part of the problem is that general theories of information [2] are very abstract and obtuse when compared with the legal views of the above authors, and we've done very little to tie these areas together to produce the necessary scientific formalisation of privacy we need.

As an example, the Privacy by Design (PbD) manifesto is being waived by many to be the commandments of privacy and following these magically solves everything. This only leads to “technical debt” and greater problems in the future. Often we find the engineers, R&D teams and the scientists excluded from, and outside of, this discussion.

I think we're missing the point what privacy really is and certainly we have little idea at this time how to effectively build information systems with inherent privacy [3] as a property of those systems. I have one initial conclusion:

WE HAVE NO UNDERLYING THEORY OF PRIVACY

We have no common definitions, common language, common semantics nor mappings between our individual worlds: legal, advocacy and engineering. Worse, in each of these worlds terminology and semantics are not always so well internally defined.

When an [software] engineer says “data collection”, "log" or "architecture", these do not mean the same to a lawyer or a consumer advocate. Indeed I don't think these terms semantically map even remotely cleanly – if at all - between these groups.

A set of PowerPoint slides with a strategy, a vision, a manifesto, good intentions, project plan or a classifications scheme mean very little and without some form of semantics are wasted, token efforts that only add to the complexity and confusion of a rapidly changing field.

We desperately need to address the problem that we must create a way of communicating amongst ourselves through which all of the internal factions within the privacy community can effectively understand each other's point of view. Only then might we even have a chance of realistically and effectively addressing the moving target of privacy issues facing end- users and businesses that rely so much on the interchange and analysis of information.

The problem with formally (or rigorously) defining anything is that it has the nasty tendency to expose holes and weaknesses in our thinking. Said holes and weaknesses are not entirely appreciated, especially when it challenges an established school of thought or a political or dogmatic balance [4].

The privacy community is constantly developing new laws and legal arguments, new sets of guidelines, manifestos and doom scenarios while the engineers are trying to address these often inconsistent and complex ideas through technical means. From the engineering perspective not only we are internally exposing flaws in database design, information system architecture and user experience but also the mismatch between engineering, legal, the world of the consumer advocate and ultimately a company's information strategy.

An information strategy needs to address everything from how the engineers develop software to how you want your company to be perceived by the consumer. How many information strategies actually address the role that information plays in a modern, global consumer ecosystem where the central concept is the collection and processing of user information? Of those, how many address the engineering and scientific levels of information?

We must take a serious retrenchment [5] step and look back at what we have created. Then we need ruthlessly carve away anything that does not either solve the communication issue within the privacy community or does not immediately serve the end-user. Reemphasizing the latter point, this explicitly means the end-user values, not what we as a privacy community might perceive to be valued by the end-user.

We must fully appreciate the close link between privacy and information, and that privacy is one of the most crosscutting of disciplines. Privacy is going to expose every single flaw in the way we collect, manage, process and use information from the user experience, as well as the application and services eco-system, and even the manner in which we conduct our system and software engineering processes and information governance. The need to get privacy right is critical not just for the existence of privacy as a technical discipline in its own right (alongside security, architecture, etc) but also for the consumer and the business.

The emphasis must be placed on the deep technical knowledge of experts in information systems – these must be the drivers and unifiers between the engineers, the lawyers, the advocates and ultimately the users. Without this deep, holistic, scientific and mathematical foundation we will not be able to sufficiently nor consistency address or govern any issues that arise in the construction of our information systems at any level of abstraction.

If the work we do in privacy does not have a scientific, consistent and formal underpinning [6] that brings together the engineers, lawyers and advocates then privacy is waste of time at best and deeply destructive to the information systems at worst.

Without this we are a disjointed community caring for ourselves and not the business or consumer and privacy becomes just a bureaucratic exercise to fulfill the notions of performing a process and metrics rendered as meaningful as random numbers.

* * *

PostScript:

Via Twitter I came across a talk given by Jean Bezivin entitled "Should we Resurrect Software Engineering?" presented at the Choose Forum in December 2012. Many of the things he presented are analogous to what is happening in privacy. He makes the point a number of times that we have never addressed the missing underlying theory of software engineering and how to really unify the various communities, fields and techniques in this area. Two points I particularly liked was that we concentrated on the solution (MDE) but never thought about the problem; the other point is the use Albert Camus' quote

<< Mal nommer les choses, c'est ajouterau malheur du monde >>

[To misname things is to add misery to the world]

A subtle hint to getting the fundamentals right: terminology and semantics!


Notes

[1] #pii2012: The Emergent Privacy-Industrial Complex

[2] Jerry Seligmann, Jon Barwise (1997) Information Flow. Cambridge University Press.

[3] I like the idea of privacy being an inherent construct in system design in much the same way that inherent safety emerged from chemical/industrial plant design

[4] A blog article discussing “mathematical catastrophes” – two that come to mind are Russel and Frege and also Russel and Gödel. Both related but the latter’s challenge to the mathematical school of thought was dramatic to say the least.

[5] A formal retrenchment step in that we not just start again but actively record what we’re backtracking on. Poppleton et.al. constructed a theory of retrenchment for software design using formal methods; the same principles apply here.

[6] If you’re still in doubt just remember that whatever decisions are made with respect to privacy, there’s a programmer writing formal, mathematical statements encoding this. Lessig’s Code is Law principle.

Measuring Privacy against Effort to Break Security

As part of my job I've needed to look at metrics and measurement of privacy. Typically I've focussed on information entropy versus, say, number of records (define "record") or other measurements such as amount of data which do not take into consideration the amount of information, that is, the content of the data being revealed.

So this lead to an interesting discussion* with some of my colleagues where we looked at a graph like this.

The y-axis is a measure of information content (ostensibly information entropy wrt to some model) and the x-axis a measure of the amount of force required to obtain that information. For any given hacking technique we can deliniate a region on the x-axis which corresponds to the amount of sophistication or effort placed into that attack. The use of the terms, effort and force here come from the physics and I think we even have some ideas on how the dimensions of these map to the security world, or actually what these dimensions might be.

So for a given attack 'x', for example an SQL inject attack against some system to reveal some information 'M', we require a certain amount of effort just for the attack to reveal something. If we make a very sophisticated attack then we potentially reveal more. This is expressed as the width of the red bar in the above graph.

One conclusion here is that security people try to push the attack further to the right and even widen it, while privacy people try to lower and flatten the curve, especially through the attack segment.

Now it can be argued that even with a simple attack, over time the amount of information increases, which brings us to a second graph which takes this into consideration:

Ignoring the bad powerpoint+visio 3D rending, we've just added a time scale (z-axis, future towards back), we can now capture or at least visualise the statement above that even an unsophisticated attack over time can reveal a lot of information. Then there's a trade-off between a quick sophisticated attack versus a long, unsophisticated attempt.

Of course a lot of this depends upon having good metrics and good measurement in the first place and that we do have real difficulties with, though there is some pretty interesting literature [1,2] on the subject and in the case of privacy some very interesting calculations that can be performed over the data such as k-anonymity an l-diversity.

I have a suspicion that we should start looking at privacy and security metrics from the dimensional analysis point of view and somewhat reverse engineer what the actual units and thus measurements are going to be. Something to consider here is that the amount of effort or force of an attack is not necessarily related to the amount of computing power, for example, brute forcing an attack on a hash function is not as forcible as a well planned hoax email and a little social engineering.

If anyone has ideas on this please let me know.


References

[1] Michele Bezzi (2010) An information theoretic approach for privacy metrics. Transactions on Data Privacy 3, pp:199-215
[2] Reijo M. Savola (2010) Towards a Risk-Drive Methodology for Priavcy Metrics Development. IEEE International conference on Social Computing/IEEE International Conference on Privacy, Security, Risk and Trust.



*for "discussion" read 'animated and heated arguments, a fury of writing on whiteboards, excursions to dig out academic papers, mathematics, coffee etc' - all great stuff :-)

A (Lego) Shuttle to Space

This must have been fun...



Thanks to El Reg for reporting on this.

The Requirement of using Formal Methods

A very interesting posting by way of LtU about the ethics, morals and necessity of using formal methods when developing software: When Formal Systems Kill: Computer Ethics and Formal Methods by Darren Abramson and Lee Pike. I'll cherry pick two quotes that stood out for me from this paper: the first being:

How to mathematically model formal systems and their environments. Unlike other engineering artifacts that are mathematically-modelled (bridge stresses, aircraft aerodynamics, etc.), many concepts in computer science are brand new, and a research challenge is simply how to mathematically model these concepts. Such concepts of course include modelling software and hardware and the environments in which they operate.

The fact that much of the work we're doing in the computer science area is brand appears curious, but if you think that in many cases the mere act of formalising something reveals how little is understood about the underlying theory. So while a new design of bridge can be built, it still uses well known and (importantly!) well understood underlying theories to verify and validate its design. Compare this to situations that arise in computer science, eg: the formalisation of policy structures for privacy, or the design of a space-based semantic web inspired, personal cloud and you quickly realise how little we know.

Privacy is an interesting issue, for example we have many groups proposing "engineering methods" for design of software systems but then failing on the actual theory.

Another case was the M3 project [1,2] and now there's a group trying to replicate that work without even attempting to learn the underlying theory...go figure...

But as the paper goes on to state, verification is truly difficult - the size, internal relationships and complexity and interactions within a computer system is immense and very fragile. The paper notes a comparison between aircraft and computer systems; while an aircraft as complex as the Airbus A380 has as many "parts" as a typical large software installation, it can survive partial failure and, more importantly, we knew how the thing would behave mechanically before it even was built.

The discussion, and source of the second quote I'll take, then moved to the ethics and morality of using formal methods as best practice:

If formal methods is a best practice of software engineering, then an engineer who does not employ it is either negligent or incompetent. But formal methods is beyond the capability of typical software engineers (otherwise, why do we need formal methods experts and researchers?) or is too time-intensive to employ, so it cannot be considered to be a best practice today.

Sadly this quote is too true, but is deeply troubling when you consider how much software is actually in production.

I feel there is an irony here operating at many levels. After years of working with the UML - a language severely criticised by some for not being formal (and vice versa by others) and techniques such as MDA, we had developers using formal mechanisms without even really knowing it. Programming languages themselves are formal languages.

Then there's the irony of agile methods which apparently eschow any use for formality and modelling (so I am told and have experienced) - the irony here is that agile methods rely upon very strict communication between the developers, testing, validation, verification and common, consistent shared understanding of what is being developed.

Our own experiences of formal methods (Alloy, B-Method [4]) being used as a best practice have been documented [3] and the results we had very extremely encouraging. Though I do remember one episode where some "uber-architect" almost had a fit that we reduced his grand designs to a single page of relatively simple specification. I guess that simplicity was overrated - suffice to say, that particular design failed for many of the reasons given the the above paper.

Overall, the paper "When Formal Systems Kill..." is an extremely important discussion on software engineering failure to embrace and utilise the very techniques that it is based upon will some very damaging and frightening outcomes.

References:

• [1] Oliver I, (2009) Information Spaces as a Basis for Personalising the Semantic Web. 11th International Conference on Enterprise Information Systems6 - 10, May 2009 Milan, Italy
• [2] Oliver Ian, Honkola Jukka, Ziegler Jurgen (2008). “Dynamic, Localized Space Based Semantic Webs”. WWW/Internet 2008. Proceedings, p.426, IADIS Press, ISBN 978-972-8924-68-3
• [3]Ian Oliver (2007). Experiences of Formal Methods in 'Conventional' Software and Systems Design. BCS FACS Christmas Workshop: Formal Methods in Industry, BCS London, UK, 17 December 2007.
• [4] Ian Oliver (2006). A Demonstration of Specifying and Synthesising Hardware using B and Bluespec. Forum on Design Languages FDL'06

Bridge Building

How to rebuild a railway bridge: