Measuring Privacy

Finally got to publish some results...more coming:

The Measurement of Privacy from Ian Oliver

On Being Privacy Risk Adverse

Being risk adverse in [IT] system development isn't always a bad idea - consider mainframe technologies which are constructed to avoid any kind of failure bringing the whole system down, or not using the latest, greatest JavaScript library for your mission-critical web development...

Risk management in privacy has come to the fore of late, especially the with publication of the NIST standard of risk management. So today's conversation about being risk adverse and how one assess risk in privacy was extremely interesting.

Consider this:

Collecting personal data (or PII) is a risky activity and therefore must be minimised as much as possible.

The definition of personal data is very weak, but it is always best to consider almost everything personal data in case it is cross-referenced with other data (which would make it personal data)

therefore

Don't collect anything. Ever.

While extreme, it shows how a misplaced understanding is many aspects, including what is risk and the nature of information (personal data) can lead to extreme situations and conclusions.

While NIST is absolutely correct in its assessment that we need proper risk management procedures, how these relate to requirements, information type and all of the other privacy ontological structure is as yet very, very weak.

In fact, terms such as personal data and PII do not come even close to being in any form usable for risk management - for this we need to go much deeper into the nature of information. For example, instead of "personal data" we could use classifications on information type and a mapping from different kinds of data (of these types) to risk metrics (note the plural). An overall risk value can then be more accurately calculated - or at least be calculated on the basic of what information we actually have.

You can read more about this approach to privacy engineering in the book: Privacy Engineering - a dataflow and ontological approach.

A Definition of PII and Personal Data

There's been an interesting discussion on Twitter about the terms PII and "personal data", classification of information and metrics.

Personally I think the terms "PII" and "personal data" are too broadly applied. Their definitions are poor at best; when did you last see a formal definition of these terms? Indeed classifying a data set as PII only comes about from the types of data inside that data set and by measuring the amount of identifiability of that set.

There now exists two problems in that a classification system underneath that of PII isn't well established in normal terminology. Secondly metrics for information content are very much defined in terms of information entropy.

Providing these underlying classifications is critical to better comprehending the data that we are dealing with. For example, consider the following diagram:

Given any set of data, each field can be mapped into one or more of the seven broad categories on the left - If we wanted we could create much more sophisticated ontologies to express this. Within each of these we can specialise more and this is somewhat represented as we move horizontally across the diagram.

Avoiding information entropy as much as possible, we can and have derived some form of metric to at least assess the risk of data being held or processed. A high 'score' means high risk and a high degree of reidentification is possible, while a low score the opposite - though not necessarily meaning that there is no risk. Each of the categories could be further weighted such as using location is twice as risky as financial data.

There could be and are some interesting relationships between the categories, for example, identifiers such as machine addresses (IPs) can be mapped into personal identifiers and locations - depending upon the use case.

I'm not going to go into a full formalisation of the function to calculate this, but a simple function which takes in a data set's fields and produces a value, say in the range 0 to 5 to state the risk of the data set might suffice. A second function to map that value to a set of requirements to handle that risk is the needed.

What about PII?  Well, to really establish this we should go into the contents of the data and the context in which that data exists. Another, rather brutal way, is to draw a boundary line across the above diagram such that things on the right-hand-side are potentially PII and those on the left not. This might then become a useful weighting metric, that if anything appears to the right of this line then the whole data set gets tagged with being potentially PII. I guess you could also become quite clever in using this division line to normalise the risk scoring across the various information classifications.

In summary, we can therefore give the term PII (or personal data) a definition in terms of what a data set contains rather than using it as a catch-all classification. This allows us then to have a proper discussion about risk and requirements.

References

Ian Oliver. Privacy Engineering: A Data Flow and Ontological Approach. ISBN 978-1497569713

How much data?!?!

I took part as one of the speakers in a presentation about analytics today; explaining how data is collected through instrumentation of applications, web pages etc, to an audience who are not familiar with the intricacies of data collection and analytics.

We had a brief discussion about identifiers and what identifiers actually are which was enlightening and hopefully will have prevented a few errors later on. This bears explaining briefly: an identifier is rarely a single field, but should be considered any one of the subsets of the whole record. There are caveats there of course, some fields can't be used as part of some compound identifier, but the point here was to emphasis that you need to examine the whole record not just individual fields in isolation.

The bulk of the talk however introduced from where data comes from. For example if we instrument an application such that a particular action is collected, then we're not just collecting an instance of that action but also whatever contextual data provided by the instrumentation and the data from the traffic or transport layer. This came as a surprise that there is so much information available via the transport/traffic layers:

Said meta-data includes location, device/application/session identifiers, browser and environment details and so on, and so on...

Furthermore data can be cross-referenced with other data after collection. A canonical example is geolocation over IP addresses to provide information about location. Consider the case where a user switches off the location services on his or her mobile device; location can still be inferred later in the analytics process to a surprisingly high-level of accuracy.

If data is collected over time, then even though we are not collecting specific latitude-longitude coordinates we are collecting data about movements of a single, unique human being; even though no `explicit' location collection seems to be being made. If you find that somewhat disturbing, consider what happens every time you pay with a credit card or use a store card.

Then of course there's the whole anonymisation process where once again we have to take into consideration not just what an identifier is, but the semantics of the data, the granularity etc. Only then can we obtain an anonymous data set. Such a data set can be shared publicly...or maybe not as we saw in a previous posting.  

Even when one starts tokenising and suppressing fields, the k-anonymity remains remarkably low, typically with more than 70% of the records remaining unique within that dataset. Arguments about the usefulness of k-anonymity notwithstanding - on the other hand it is one of the few privacy metrics we have,

So, the lesson here is rather simple, you're collected a massive amount more than you really think.

The next surprise was how tricky or "interesting" this becomes when developing a privacy policy that contains all the necessary details about data collection, meta-data collection, traffic data collection; and then the uses to which that data is put, whether it is primary or secondary collection and so on.

Foundations of Privacy - Another Idea

This got triggered by a post on LinkedIn about what a degree in privacy might contain. I've certainly thought about this before, at least in terms of software engineering, and even have a whole course that could be taken over a semester ready to go.

Aside: CMU has the "World's First Privacy Engineering Course": a Master of Science in Information Technology—Privacy Engineering (MSIT-PE) degree. So, close, but a major university here in Finland turned down the chance to create something similar a few years back...

That aside, I've been wondering about how to present they various levels of things we need to consider to properly define privacy and put it on strong foundations. Though in the guise of information theory we already have this, though admittedly Shannon's seminal work from the 1930's is maybe a little too deep. On the other hand understanding concepts such as channels, entropy are fundamental building blocks, so maybe they should be there along with privacy law - now that would make some course!

Even just sketching out areas to present and what might be contained therein...how about this, even if a linear map from morality to mathematics is too constraining?

There are missing bits - we still have a  semantic gap between the "legal world" and the "engineering world"; parts that I'm hoping that things such as the many conferences, academic works and books such as the excellent Privacy Engineer's Manifesto and Privacy Engineering will play a role in defining. Maybe the semantic gap goes away once we start looking at this...is there even a semantic gap? 

However, imagine for a moment starting anywhere in this stack and working up and down and keeping everything linked together in the context of privacy and information security. Imagine seeing the link between EU privacy laws and type theory, or between the construction of policies and entropy, the algebra of HIPAA, a side course in homotopy type theory and privacy...maybe with that last one I'm getting carried away, but, this is exactly what we need to have in place.

Each layer provides the semantics to the layer above - what do our morals and ethics means in terms of formalised laws, what do laws mean in terms of policies, what do policies mean in terms of software engineering structures, and down to the core mathematics and algebras of information.

Privacy and privacy engineering in particular almost has everything: law, algebra, morals, ethics, semantics, policy, software, entropy, information, data, BigData, Semantic Web etc etc etc. Furthermore, we have links to areas such as security, cryptography, economic theory etc!

Aren't these the very things any practitioner of privacy (engineering) should know, or at least have knowledge of? Imagine if lawyers understood information theory and semantics, and, software engineers understood law? 

OK, so there might be various ways of putting this stack together, competing theories of privacy etc, but that would be the real beauty here - a complete theory of privacy from the core mathematics through physics, computation, type theory, software engineering, policies, law and even ethics and morals.

But again, no more naivety, no more terminological or ontological confusions, policies and laws being traceable right down to the computation structures and code. Quite a tall order, but such a course bringing all these together really would be wonderful...

And wouldn't that be something!

Classifying Data Flow Processes

In previous posts (here, here, here and especially here) we presented various aspects for classifying data nominally being sent over data flows or channels between processes. Now we turn to the processes themselves and tie the classification of channels together.

Consider the data flow shown here (from the earlier article on measurement, from where we take the colour scheme of the flows) where we show the movement of, say, location information:

Notice how data flows via a process called "Anonymisation" (whatever that word actually means!) to the advertiser. During the anonymisation the location data is cleaned to a degree that business or our privacy policy allows - such boundaries are very context dependent.

This gives the first type of process, one that reduces the information content.

The other type of processes we see are those that filter data and those that combine or cross-reference data.

Filtering is where data is extracted from an opaque or more complex set of information into something simpler. For example if we feed some content, eg: a picture, then we might have processes that filter out the location data from said pictures.

Cross-referencing means that two or more channels are combined to produce a third channel containing information from the first two. A good example of this is geo-locating IP addresses which takes in as one source an IP address and another a geographical location look-up table. Consider the example below:

which combines data from secondary and primary sources to produce reports for billing, business etc.

In the above case we probably wish to investigate the whole set of processes that are actually taking place and make a considerable amount of decomposition of the process.

When combined with the classifications on the channels, particularly information and security classes we can make some substantial reasoning. For example, if there is a mismatch in information content and/or security classifications then we have problems; similarly if some of these are transported over insecure media.

To summarise, in earlier articles we explained how data itself may be classified, and here how processes may be classified according to a simple scheme:

• Anonymising
• Filtering
• Cross-Referencing

In a later article I'll go more into the decomposition and refinement of channels and processes.

Measurement and Metrics for Information Privacy

We have already discussed an analogy for understanding and implicitly measuring the information content over a data channel. The idea that information is an "infectious agent" is quite a powerful analogy in the sense that it allows us better to understand the consequences of information processing and the distribution of that data better, viz:

• Any body of data which is containing certain amounts and kinds of sensitive data we can consider to be non-sterile
• Information which is truly anonymous is sterile
• Mixing two sets of information produces a single set of new information which is as at least as unclean as the dirtiest set of data mixed, and usually more so!
• The higher the security classification the dirtier the information

Now we can using our information classification earlier introduced we can further refine our understanding and get some kind of metric over the information content.

Ley us classify information content into seven basic categories: financial, health, location, personal, time, identifiers and content. Just knowing what kinds of data are present as we have already discussed gives us a mechanism to pinpoint where more investigation is required.

We can then go further an pick out particular data flows for further investigation and then map this to some metric of contamination:

For example, transporting location data has a particular set of concerns, enough to make any privacy professional nervous at the least! However if we examine a particular data flow or store we can evaluate what is really happening, for example, transmitting country level data is a lot less invasive than highly accurate latitude and longitude.

Now one might ask why not deal with the accurate data initially? The reasons are that we might not have access to that accurate, field-level data, we might not want to deal with the specifics at a given point in time, specific design decisions might not have been made etc.

Furthermore, for each of the seven categories we can give some "average" weighting and abstract away from specific details which might just complicate any discussion.

Because we have a measure, we can calculate and compare over that measurement. For example, if we have a data channel carrying a number of identifiers (eg: IP, DeviceID, UserID) we can take the maximum of these as being indicative of the sensitivity of the whole channel for that aspect.

We can compare two channels, or two design decisions, for example, a channel carrying an applicationID is less sensitive (or contaminated) than one carrying device identifiers.

We also can construct a vector over the whole channel composed out of the seven dimensionsb above to give a further way of comparing and reasoning about the level of contamination or sensitivity:

| (FIN=0,HLT=0,LOC=10,PER=8,TIM=3,ID=7,CONT=0) | 
   < 
| (FIN=3,HLT=2,LOC=4,PER=4,TIM=2,ID=9,CONT=2) |

for some numerical values gien to each category. Arriving at these values will be specific to a given wider context and then the weighting given to each, but there is one measure which can be used to ground all this, and that is of information entropy, or, how identifiying the contents are to a given, unique human being. A great example of this is given at the EFF's Panopticlick pages.

We've only spoken about a single data flow at the moment, however the typical scenario is for reasoning over longer flows, for example, we might have our infrastructure set up as below*

In this example we might look at all the instances where AppID and Location are specified together and use a colour coding such that:

• Black: unknown/irrelevant
• Red: high degree of contamination, both AppID and Location unhashed and accurate respectively
• Yellow: some degree of contamination, AppID may be hashed(+salt?) or Location at city level or better
• Green: AppID randomised over time, hashed, salted and Location at country level or better

Immediately readable from the above flow are our points of concern which need to be investigated, particular the flows from the analytics processing via the reports storage and to direct marketing. It is also easy to see that there might be a concern with the flow to business usages, what kinds of queries ensure that the flow here is less contaminated than the actual reports storage itself?

There are a number of points we have not yet discussed, such as that some kinds of data can be transformed into a different type. For example some kinds of content such as pictures inherently contain device identifiers, locations etc. Indeed the weighting for some categories such as content might be very much higher than that of identifiers for example - unless the investigation is made. Indeed it does become almost a trivial exercise for some to explicitly hide sensitive information inside opaque data such as images and not declare then when a privacy audit is made.

To summarise, we've a simple mechanism for evaluating information content over a system in quantative terms, complete with a refinement mechanism that allows us to work at many levels of abstraction depending upon situation and context. Indeed, what we are doing is explcitly formalising and externalising our skills when performing such evaluations, and through analogies such as "infection control" providing means for easing the education of professionals outside of the privacy area.

*This is not indicative of any system real, living or dead, but just an example configuration.

Information as an Infectious Agent

Operating theatres are split into two parts:

• the sterile field
• the non-sterile surroundings

Any non-sterile item entering the sterile field renders it non-sterile; and stringent efforts and protocols [1,2] are made to ensure that this does not happen.

The protocols above extend via simply analogy [3,4] to information handling and information privacy.

• Any body of data which is containing certain amounts and kinds of sensitive data we can consider to be non-sterile - assume for a moment that certain bits and bytes are infectious (great analogy!).
• Everyone working with information is required to remain sterile and uncontaminated.
• Information which is truly anonymous is sterile
• Mixing two sets of information produces a single set of new information which is as at least as unclean as the dirtiest set of data mixed, and usually more so!
• The higher the security classification the dirtier the information

We can extend this latter point to specific information types, eg: location, personal data, or certain kinds of usages and purposes, eg: data for advertising or secondary data and so on.

Extending our analogy further we can protect the sterile field in two ways:

• ensuring that everyone in contact with the sterile field is sterile
• ensuring that the equipment entering the sterile field is sterile

Basic data handling techniques are essential:

• If two sets of data are to be mixed then ensure that the mixing occurs not in-situ but by generating a third data set kept separate from the two input sets
• Data can be made more sterile by removing information content. But, be warned that certain kinds of obfuscation are not effective, eg: hashing or encryption of fields might just hide the information content of that field but not the information content of the whole data set [3]
• Keep sterile and non-sterile data-sets apart, physically if possible
• Ensure that sterile and non-sterile data-sets have differing access permissions. Ideally different sets of people with access
• Clean up after yourself: secure data deletion, overwriting of memory, cache purges etc.

From a personnel point, in surgery precautions are made through restricting the persons inside the sterile field and even outside of this, basic precautions are taken in terms of protective clothing etc. While surgical attire might be overkill for office environments, the analogy here is that personnel with access to data have received the correct training and are aware of what data they can and can not use for various purposes.

In a surgical environment, everything entering and leaving the sterile field is checked and recorded. In an information systems environment this means logging of access so that when a breach of the sterile field occurs the route of the pathogen and its nature can be effectively tracked and cleaned.

References

[1] Infection Control Today - August 1, 2003 : Guidelines for Maintaining the Sterile Field
[2] Infection Control Today - November 7, 2006 - Best Practices in Maintaining the Sterile Field

[3] What has surgery got to do with information privacy?

[4] Flying planes, Surgery and Privacy - the presentation

Analysing Data Flow Models

In the previous post we introduced a pattern for the data-flows in and out of an application such as those found on your mobile phone or tablet. In this posting I want to expand on this pattern and explore various annotations to help us reason about how our application treats the information flowing through it.

Let's first introduce an example application, a photo sharing application. This app allows you to select a photo on your device and upload it to a social media account. Here's what it looks like to the user on a mobile device with rounded corners (as opposed to square corners which none of them seem to have):

It looks innocent enough but as we know there are many threats to user privacy even in the most innocent looking of places. So let's model what is really happening behind the scenes. We understand so far a number of things: the user supplies content and credentials for the services, these are stored locally for convenience, the app adds meta-data to the picture before uploading and sends information about the app's behaviour to the inventor of the app. We might then construct the following model:

On each of the dataflows we have noted the kind of information transported over those channels and the mechanism of communication. We also note our presumed trust boundary.

What this shows clearly is where data is flowing and by what means. We have for now skipped over the specific or precise meanings of some things hoping that the terms we have used are self-explanatory.

But, now we have formally written we can focus the discussion on specific aspects of the application, for example:

• What mechanisms are being used to store the user ID and password in the "operating system"? Is this storage secure and sandboxed? I.e. how do we increase the area of trust boundary?
• Are the communication mechanisms from the app to the social media and inventor appropriate?
• What infrastructure information is implicitly included over these channels, for example, IP addresses, user agent strings [1] etc?
• Does the app have authorisation to the various channels?
• What is the granularity of the Location data over the various channels?
• What information and channels are considered primary and which secondary?
• Is the information flowing to the inventor appropriate and what is the content?
• What about the EXIF data embedded in the picture?

Generating a formal list of questions from the above is relatively easy and this is exactly how we proceed.

The next questions which follow are related to how we can reduce the amount of information without compromising the application functionality and business model? For example:

• can we reduce the granularity of the Location sent to the social media systems to, say, city level or country level?
• can we automatically remove EXIF data?
• do we allow the app to work if the operating system's location service is switched off or the user decides not to use this?

And so on...Finally  we get down to the consent related questions

• What does the user see upon first-time usage of the app? What do they have to agree to?
• Do we tell the user what underlying services such as GPS we're using as part of the application
• Secondary data collection opt-out
• For what reason is the data being collected over both primary and secondary channels

And so on again.

What we have done is set the scene, or, circumscribed what we need to investigate and decide upon. Indeed at some level we even have the means to measure the information content of the application and even the extent the consents by implication; and if we can measure then we have a formal mechanism to decide whether one design is "better" than another in terms of privacy.

In the following articles I'll discuss more about the classification mechanisms (information, security, usage, purpose, provenance) and other annotations along with the detailed implications of these.

Notes

[1] User agent strings are very interesting...just ask Panopticlick.

Systems Safety - Defining Moments

As I've been concentrating on "safety improvements", or at least techniques for the improvement of system I've tended to concentrate on four areas:

• Aviation
• Industrial
• Medical (specifically surgical)
• Software Engineering (specifically information privacy)

The parallels between theses areas when it comes to what each define as "safety" and the techniques should be obvious. However the question remains what actually triggered each of these respective areas to take a more inherent safety approach and then what will trigger a similar approach with regards to information safety?

 

Above Diagram Key: Y-Axis: relative degree of safety embedded into that discipline, X-Axis, year of time since seminal incident.

Aviation safety's seminal moment was the 1935 crash of a Boeing Model 299 aircraft during a presentation flight. Instead of blaming the pilots, effort was made to understand the causes of the accident and develop techniques to help prevent similar accidents in the future.

For industrial safety the seminal moment was the 1974 Flixborough Disaster in the UK. This resulted in work on the design of industrial plants and the development of the notion of "inherent safety".

Surgical safety has quite a long tradition especially with the development of anaesthetic safety from the 1960s and the introduction of a proper systems approach. However anesthetists seem not to feature prominently as surgeons and doctors so the fame would probably go to Peter Pronovost et.al. for the Central Line checklist. This was probably one of the major contributors to the WHO Surgical Safety Checklist discussed in detail in Atul Gawande's book The Checklist Manifesto which brings together much of the above incidents. 

If you're still in doubt maybe Atul Gawande's article in the New Yorker magazine entitled The Checklist: If something so simple can transform intensive care, what else can it do? (Dec 10, 2007) might help.

Getting back to the crux of this article, what is the incident that will cause the wholesale change in attitudes and techniques to software engineering that instills such a sense of discipline that we can eradicate errors to such a degree that we could compare ourselves favourably with other disciplines?

The increasingly frequent hacking and information leaks? The NSA wiretapping and mass surveillance? Facebook and Google's privacy policies? None of these have had any lasting effect upon the very core of software engineering if any at all. Which either means that we place such low value on the safety of our information or that the economics of software are so badly formulated in society that the catastrophe would have to be so huge that it would have to cause societal change?

Interestingly, in software engineering and computer science we're certainly not short on techniques for improving the quality and reliability of the systems we're developing: formal methods (eg: Alloy, B, Z, VDM etc), proof, simulation, testing, modelling (in general). What we probably lack is the simplicity of a checklist to guide us through the morass of problems we encounter. In this last respect, this is why I think we're more like surgeons that modern day aviators; or, maybe some of us are like the investigators to the 1935 Boeing crash and other aviation heroes learning their trade?

Data Collection

Anyone who cares about privacy will tell you that data collection is bad...REALLY BAD, yet without data collection none of the services we use and need would work (there's an economic argument there). Indeed most of the issues around data collection seems to be very emotive in nature and usually end up in the "data collection is bad/must be minimsed panic".

First to understand data collection we must understand from where data is collected. First of all we need to understand the relationship between a client and a server via some infrastructure. This is described in an earlier article which talks about primary and secondary data.

Generally the case we mainly worry about in privacy is where the client is a human interacting with an app or browser via some device, eg: a mobile phone.

The end-user (customer or human) will provide some information required to fulfill the functions of the service required, eg: posting a photograph to Flickr with some description and their login details.

So far this seems reasonable - afterall, a user is posting a photograph to their account with a description. The service might then perform an number of other tasks

• Set the current time and date of upload
• Extract EXIF information from the photograph, including (but not limited to):
• date and time of the shot
• location information
• camera information
• free text strings, eg: copyright/ownership information

All of the above comes under the notion of data collection and all of the above is information provided by the user of the service. One can argue that the user didn't know about the EXIF contents of the photograph and surprisingly few people actually realise how much information is embedded into the picture.

Further operations can be performed such as facial recognition and linking this to persons who have been tagged in other photographs etc. For example, this is a service provided by Facebook.

Additionally the service maybe collecting secondary information about the usage of the service, for example, how much time was spent using a particular page, what the UI/UX flow was and so on.

Then there's the information collected from the infrastructure. This includes information provided by the browser or app, typically in the form of browser identification strings, application identifiers, device identifiers, and from the service infrastructure including such information as source IP address, contents of the API call, time and date of interaction, error codes etc.

When we talk about privacy we typically end up focussing on the primary and secondary information sets. The former set to decide whether that information is necessary for the service to function and provide the facilities the user requires and the second set to decide whether this information needs to be explicitly collected or not. Rarely do we even investigate the information collected by the infrastructure which can be used both to recreate primary and secondary data sets. Indeed one of the major targets for any hacker are the infrastructure logs themselves.

When we talk about data collection we must consider the following data sets:

Primary, secondary and infrastructure are described as above with the addition that we split primary into explicit and implicit where the former denotes that information which is explicitly understood by the end-user and implicit including that information that might be hidden, eg: EXIF. Combined together we produce the Total Information Set which when linked together by simple association becomes the Total Deduced Information Set.

Another way of looking at this is in a more Venn diagrammatic manner:

Where the labels are abbreviations of the corresponding labels in the first diagram and colours for emphasis.

It must be noted that some intersections might not properly exist, for example the union of EP and Sec, and others only in certain contexts, for example, whether IP is a proper superset of EP or not.

Now we have explicitly modelled data collection, we can now start tackling questions such as what does minimisation of data collection actually mean and then start looking at what the processing and extraction functions over that data might actually look like.

On The Naivety of Privacy

Recent events regarding privacy and the internet have left me wondering if we are being somewhat naïve. We are starting to see a slew of new laws, strategies and technologies for protecting our privacy in what is effectively a public space. The end-user however is not, as far as I can tell, really getting the benefit of this - indeed if anyone is it is the emerging privacy-industrial complex [1] as some have written.

It is utterly naïve to believe that laws, strategies, intentions, grand speeches, certifications, automatic filtering, classification iconography and so on make for better end-user privacy. The more we do this the more confused we become, and simultaneously we lose sight of what we're really trying to achieve. Spare a thought for the poor end-users.

There is a great deal that is misunderstood or not known by privacy advocates about how the internet, computers and information systems work - I fear in a lot of cases either some don't want to understand because it takes them outside of their comfort zone, or the semantic gap between the engineers and the legal/advocacy side is too great and that bridging this gap is extraordinarily difficult for both parties.

I worry about our lack of formality and discipline, possibly in equal quantities. We – the privacy community – lack these aspects to really understand and accept the fundamentals of our area and how to apply this to the information systems we are trying to protect. In some cases we are actively fighting against the need to scientifically ground our chosen area.

We must take a moment to think and understand what problem we are really trying to solve. The more philosophical works by Solove and Nissenbaum address the overall concept of privacy. I'm not sure that the implications of these are really understood. Part of the problem is that general theories of information [2] are very abstract and obtuse when compared with the legal views of the above authors, and we've done very little to tie these areas together to produce the necessary scientific formalisation of privacy we need.

As an example, the Privacy by Design (PbD) manifesto is being waived by many to be the commandments of privacy and following these magically solves everything. This only leads to “technical debt” and greater problems in the future. Often we find the engineers, R&D teams and the scientists excluded from, and outside of, this discussion.

I think we're missing the point what privacy really is and certainly we have little idea at this time how to effectively build information systems with inherent privacy [3] as a property of those systems. I have one initial conclusion:

WE HAVE NO UNDERLYING THEORY OF PRIVACY

We have no common definitions, common language, common semantics nor mappings between our individual worlds: legal, advocacy and engineering. Worse, in each of these worlds terminology and semantics are not always so well internally defined.

When an [software] engineer says “data collection”, "log" or "architecture", these do not mean the same to a lawyer or a consumer advocate. Indeed I don't think these terms semantically map even remotely cleanly – if at all - between these groups.

A set of PowerPoint slides with a strategy, a vision, a manifesto, good intentions, project plan or a classifications scheme mean very little and without some form of semantics are wasted, token efforts that only add to the complexity and confusion of a rapidly changing field.

We desperately need to address the problem that we must create a way of communicating amongst ourselves through which all of the internal factions within the privacy community can effectively understand each other's point of view. Only then might we even have a chance of realistically and effectively addressing the moving target of privacy issues facing end- users and businesses that rely so much on the interchange and analysis of information.

The problem with formally (or rigorously) defining anything is that it has the nasty tendency to expose holes and weaknesses in our thinking. Said holes and weaknesses are not entirely appreciated, especially when it challenges an established school of thought or a political or dogmatic balance [4].

The privacy community is constantly developing new laws and legal arguments, new sets of guidelines, manifestos and doom scenarios while the engineers are trying to address these often inconsistent and complex ideas through technical means. From the engineering perspective not only we are internally exposing flaws in database design, information system architecture and user experience but also the mismatch between engineering, legal, the world of the consumer advocate and ultimately a company's information strategy.

An information strategy needs to address everything from how the engineers develop software to how you want your company to be perceived by the consumer. How many information strategies actually address the role that information plays in a modern, global consumer ecosystem where the central concept is the collection and processing of user information? Of those, how many address the engineering and scientific levels of information?

We must take a serious retrenchment [5] step and look back at what we have created. Then we need ruthlessly carve away anything that does not either solve the communication issue within the privacy community or does not immediately serve the end-user. Reemphasizing the latter point, this explicitly means the end-user values, not what we as a privacy community might perceive to be valued by the end-user.

We must fully appreciate the close link between privacy and information, and that privacy is one of the most crosscutting of disciplines. Privacy is going to expose every single flaw in the way we collect, manage, process and use information from the user experience, as well as the application and services eco-system, and even the manner in which we conduct our system and software engineering processes and information governance. The need to get privacy right is critical not just for the existence of privacy as a technical discipline in its own right (alongside security, architecture, etc) but also for the consumer and the business.

The emphasis must be placed on the deep technical knowledge of experts in information systems – these must be the drivers and unifiers between the engineers, the lawyers, the advocates and ultimately the users. Without this deep, holistic, scientific and mathematical foundation we will not be able to sufficiently nor consistency address or govern any issues that arise in the construction of our information systems at any level of abstraction.

If the work we do in privacy does not have a scientific, consistent and formal underpinning [6] that brings together the engineers, lawyers and advocates then privacy is waste of time at best and deeply destructive to the information systems at worst.

Without this we are a disjointed community caring for ourselves and not the business or consumer and privacy becomes just a bureaucratic exercise to fulfill the notions of performing a process and metrics rendered as meaningful as random numbers.

* * *

PostScript:

Via Twitter I came across a talk given by Jean Bezivin entitled "Should we Resurrect Software Engineering?" presented at the Choose Forum in December 2012. Many of the things he presented are analogous to what is happening in privacy. He makes the point a number of times that we have never addressed the missing underlying theory of software engineering and how to really unify the various communities, fields and techniques in this area. Two points I particularly liked was that we concentrated on the solution (MDE) but never thought about the problem; the other point is the use Albert Camus' quote

<< Mal nommer les choses, c'est ajouterau malheur du monde >>

[To misname things is to add misery to the world]

A subtle hint to getting the fundamentals right: terminology and semantics!


Notes

[1] #pii2012: The Emergent Privacy-Industrial Complex

[2] Jerry Seligmann, Jon Barwise (1997) Information Flow. Cambridge University Press.

[3] I like the idea of privacy being an inherent construct in system design in much the same way that inherent safety emerged from chemical/industrial plant design

[4] A blog article discussing “mathematical catastrophes” – two that come to mind are Russel and Frege and also Russel and Gödel. Both related but the latter’s challenge to the mathematical school of thought was dramatic to say the least.

[5] A formal retrenchment step in that we not just start again but actively record what we’re backtracking on. Poppleton et.al. constructed a theory of retrenchment for software design using formal methods; the same principles apply here.

[6] If you’re still in doubt just remember that whatever decisions are made with respect to privacy, there’s a programmer writing formal, mathematical statements encoding this. Lessig’s Code is Law principle.

Measuring Privacy against Effort to Break Security

As part of my job I've needed to look at metrics and measurement of privacy. Typically I've focussed on information entropy versus, say, number of records (define "record") or other measurements such as amount of data which do not take into consideration the amount of information, that is, the content of the data being revealed.

So this lead to an interesting discussion* with some of my colleagues where we looked at a graph like this.

The y-axis is a measure of information content (ostensibly information entropy wrt to some model) and the x-axis a measure of the amount of force required to obtain that information. For any given hacking technique we can deliniate a region on the x-axis which corresponds to the amount of sophistication or effort placed into that attack. The use of the terms, effort and force here come from the physics and I think we even have some ideas on how the dimensions of these map to the security world, or actually what these dimensions might be.

So for a given attack 'x', for example an SQL inject attack against some system to reveal some information 'M', we require a certain amount of effort just for the attack to reveal something. If we make a very sophisticated attack then we potentially reveal more. This is expressed as the width of the red bar in the above graph.

One conclusion here is that security people try to push the attack further to the right and even widen it, while privacy people try to lower and flatten the curve, especially through the attack segment.

Now it can be argued that even with a simple attack, over time the amount of information increases, which brings us to a second graph which takes this into consideration:

Ignoring the bad powerpoint+visio 3D rending, we've just added a time scale (z-axis, future towards back), we can now capture or at least visualise the statement above that even an unsophisticated attack over time can reveal a lot of information. Then there's a trade-off between a quick sophisticated attack versus a long, unsophisticated attempt.

Of course a lot of this depends upon having good metrics and good measurement in the first place and that we do have real difficulties with, though there is some pretty interesting literature [1,2] on the subject and in the case of privacy some very interesting calculations that can be performed over the data such as k-anonymity an l-diversity.

I have a suspicion that we should start looking at privacy and security metrics from the dimensional analysis point of view and somewhat reverse engineer what the actual units and thus measurements are going to be. Something to consider here is that the amount of effort or force of an attack is not necessarily related to the amount of computing power, for example, brute forcing an attack on a hash function is not as forcible as a well planned hoax email and a little social engineering.

If anyone has ideas on this please let me know.


References

[1] Michele Bezzi (2010) An information theoretic approach for privacy metrics. Transactions on Data Privacy 3, pp:199-215
[2] Reijo M. Savola (2010) Towards a Risk-Drive Methodology for Priavcy Metrics Development. IEEE International conference on Social Computing/IEEE International Conference on Privacy, Security, Risk and Trust.



*for "discussion" read 'animated and heated arguments, a fury of writing on whiteboards, excursions to dig out academic papers, mathematics, coffee etc' - all great stuff :-)

Data-Flows and Measurement of Expectation of Privacy

I've been in a workshop all day about privacy with a mixed audience of legal, marketing and technical people; and its quite interesting to see that we starting to have some convergence on that privacy is more about information itself, the flow of information and the usage of that information within the context of those flows rather than the usual discussion about how to prevent collection of data.

There is relatively little wrong - given the correct context - with data collection, and indeed in many cases it is ineviatable, eg: web server or service access logs. The usage of these logs for system monitoring is the typical scenario, which is a necessary function of actually running those infastructures. The main point here is really aimed at secondary data collection or behavioural data collection scenarios.

So that aside for a moment, we've come to the obvious conclusion that security is a necessary base for privacy, which in turn is a necessary base for trust. We've also discussed the notion of rights and what rights a consumer has over their data, or more correctly, their information.

Which all brings me back to that most of the discussions are touching on the need for an understanding of the flow and measure of information. How do we measure, what do we measure, how much information, is there too much information etc?

Putting this in the context of information management, ontologies/taxonomies of information and data-flow we have the beginnings of a rather elegant framework for understanding the flow of information from this perspective. Sounds close to Nissenbaum's hypothesis on privacy and expectations which is very nice - which is something I've written on before and I guess some of the things here is a development of some thoughts there...

For me this means that some ideas I’ve had of information classification, dimensional analysis and measures (metrics even) are starting to coalesce nicely...quite exciting.

In a panel session a discussion was held on the rights and relationships of privacy to the consumer and started to emphasise on the expectation of privacy based in various scenarios: placing data in the cloud, driving on a public highway and in relation to the latter the case with the US government's regarding the placement of GPS trackers on peoples' cars without their knowledge.

We can construct a data-flow model of this:

          

Over each flow we can measure the amount, sensitivity and type of information - I have no idea what this "number" or even the structure of that "number" might look like, though I do believe that it is certainly measureable, ie: we can take two values and compare them.

A person then assigns or has an expectation of privacy in various situations, if the data-flow exceeds that then there is a privacy issue. So, using some “arbitrary” values for the measures, we might have expectations ‘E’ for each flow:

• E(Person->Cloud) is 7
• E(Person->Highway) is 3
• E(Highway->Government) is 2

The higher the number, the greater amount of information a user is willing to tolerate being communicated over that data-flow.

Then at some point in time’t’ the actual measure ‘M’ of information, maybe something like

• M_t1(Person->Cloud) = 5
• M_t1(Person->Highway) = 2
• M_t1(Person->Cloud) = 4

If for some data-flow ‘d’, at a point in time ‘t’, M_t(d)>E(d) then we have a problem regarding the amount of information being transmitter is greater than the expectations of the user.

Aside: yes, I know using integers to denote amount is fairly naïve, but I’m just trying to get a point across more than anything – I think the structure we’d be working with is some horrible multi-dimensional, tensor/spinor monster….

While the current laws tend to focus on the fact that anything ‘in public’ is ‘public’, Solove, Nissenbaum, Acquisti and others have noted that what happens in public is not always necessarily. As shown in the data-flow above, a person's expectation of privacy towards some cloudified service environment, eg: Google, Nokia etc is very different to their expectation of privacy when driving in their car on public roads. Similarly the information flow between public roads and the government, eg: traffic cameras etc has certain expectations of privacy.

When we have information flow over more than one individual flow, for example, what is the user's expectation of privacy when information about their driving on a public road flows to the government? The case with GPS trackers has shown that there are expectation limits that are different from the individual expectations within the individual flows, for example:

• E(Person->Highway->Government) = 1

What this eventually leads to is that as data-flows get longer and involve more participants the expectation of privacy increases, but in reality beyond one or two steps the visibility of the data-flow diminishes to the user, for example, to where does Google or Facebook send or sell their data? Also how. and could this value be calculated from each of the individual flows? I can imagine that we might even see some kind of power law operating over this too…

Many other questions arise, how do we measure information content – at least in terms of the above channels? What is an information channel? To conclude for the moment, it does appear that we can relatively easily define how to how these measures might behave over a data-flow, the question now remains – and this is the really interesting question – is how to actually construct the measure itself.

Research, Productivity and Individualism

Brilliant article from the New York Times:

The Rise of the New Groupthink

By SUSAN CAIN

Published: January 13, 2012

SOLITUDE is out of fashion. Our companies, our schools and our culture are in thrall to an idea I call the New Groupthink, which holds that creativity and achievement come from an oddly gregarious place. Most of us now work in teams, in offices without walls, for managers who prize people skills above all. Lone geniuses are out. Collaboration is in

A few years ago at Nokia Research I took part in research to understand from where we researchers/scientists/engineers found inspiration and the environments in which we did the best work. Partly it was research to justify the move to a "collaborative working environment" or removal of the offices and stuck us all in an open area depending upon your point of view.

After many interviews and an outbreak of common sense the allocation of offices remained and productivity continued.

At least in the areas I have worked, individualism and office privacy are critical to the thinking process. There are times when you must "socialise" and present ideas but this is done in a number of forums: the small (2-3 persons) group, the larger (3+) group - the latter useful for semi-formal presentations and then the formal lecture (7+) persons where interaction is more strictly controlled. After, certainly the small group sessions, retreat to a quiet, private and personal space is critical for the whole thinking, development and inspiration process; from the above article talking about the emphasis (at least in the press) on the collaborative working environment at Apple:

The story of Apple’s origin speaks to the power of collaboration. Mr. Wozniak wouldn’t have been catalyzed by the Altair but for the kindred spirits of Homebrew. And he’d never have started Apple without Mr. Jobs. 

But it’s also a story of solo spirit. If you look at how Mr. Wozniak got the work done — the sheer hard work of creating something from nothing — he did it alone. Late at night, all by himself.

NB: emphasis mine in the above

I remember one of the outcomes of the working environments research was that the ideas came in those solitary moments - occasionally 3 o'clock on a rainy, cold Sunday morning. To me the brain requires more of these solitary times without interruption in order to process the accumulated information and churn that into "knowledge".

The article also raised, amongst others, two other points:

Brainstorming is usually a bad idea, and this I fully agree for all the reasons stated in the article.

The second point is the emphasis - as seen in many job descriptions - for an "outgoing, charismatic, team-player"...admit that you like reading books, walking alone or being individualistic and you can forget the job. The sorts of people attracted by these (and I've been requested on more than one occasion to add that as part of a job description) do not work well, nor have good productivity and research skills in the environments and for the works I (as a computer scientist/mathematician) participate in.

Fortunately the team I work with now is a collection of highly skilled individuals who understand how to work together and how each of us needs that freedom, privacy and isolation to do out best work.

Beauty in Mathematics

Sometimes when "doing" mathematics you end up with something that succinctly captures what you mean and presents it in a rather beautiful way. Now while beauty in mathematics has many examples - in particular Euler's Identity and in my opinion the sub-object classifier from category theory (it even has a facebook page) - formal specification is not really one area where this has been seen.

Here's a quick diagram from a paper I've been collaborating on with the Visual Modelling Group in Brighton, I personally think that this very succinctly states the relationship between information entropy and various classifications - at least in the framework we're using...

[0,1] is the 'set' of Reals and the precise definition of infoMeasure not strictly specified but the capturing of the mapping between various IP addresses and their geolocation mapping to various levels of detail of accuracy, and from there to a measurement in terms of entropy is still rather elegant.

Returning back to Euler's Identity...funny how Euler's work has an inherent beauty in it, especially when you consider Euler's Identity was considered the most beautiful mathematical expression and Eulerian circles the basis of the diagrammatic notation used above.