Where are all the Information Scientists?

What happened to all the scientists and engineers with a real deep understanding of what information is? We're now apparently firmly in the era of Big Data and yet most of the Big Data implementations seem to stop be building large Hadoop clusters and running modified versions of the canonical word counting programs over log files from Apache.

I had the pleasure of working with a group of people who were blessed with PhDs in mathematics, physics and statistics and yet they complained bitterly about the lack of good, interesting and relevant data in order to properly ply their skills and generate valuable insights.

I know of one person who is utilising some quite cool natural language processing over massive data sets to extract all kinds of context, and yet no-one seems to be interested in what this is achieving.

The current Gartner Hype Curve for Big Data has us on the edge of the precipice into the Trough of Disillusionment and once again when the negative press starts everyone will be surprised. Worse is that adopters of Big Data and analytics will not question why it went wrong but rather blame Big Data for not delivering.

One of the reasons why Big Data is heading for this "failure" is because we do not understand, or more accurately want to understand our data. We do not want to put the investment into understanding the subtle and complex interactions between the various aspects of our data, the classification structures, the processing etc.

Indeed I find much of the current trend to NoSQL as being an excuse for ignoring the very basics of understanding data. Apparently SQL is bad: I'm told that SQL means static table structures, fixed schema, inflexible query languages and all that relational theory stuff about data organisation is obsolete. and irrelevant. Multi terabyte "traditional" databases are uncool, cloudified Linux clusters in AWS running Hadoop and canonical word counting programs over web log files is COOL? At one level we've just made OS/360 and COBOL more complicated without doing much more than was possible 30 years ago.

Have we really forgotten how to manage and construct our data so that it becomes information? Or, as I also worry about, have we lost the ability to understand what our data could tell us and been blinded by the hype behind a technology? Not that it hasn't happened before of course.

Much of our emphasis is on building products and getting cool apps to users; sometimes we get data out of these apps either by primary or secondary means. Surprisingly little thought is placed on what data and how we could really understand that data and even less on the interactions between different datasets and what that could tell us.

Rather than understanding the context in which our information exists we've become obsessed with generating abstract counts of things which translate easily into bar charts on a PowerPoint slide. We're failing at extracting the deeper context.

It's fine to learn that an application is being used by one million users this week and two million the next, but does this really tell you about how people are using that app, who is using that app, why, where and what for? Even more importantly we don't even consider the inverse of the questions, why aren't people using my app?

Consider air miles or store cards - when was the last time an airline or supermarket contacted you to ask you why or why not you are flying/shopping with them? I can guarantee that the airline or supermarkets has number of how many things were bought or flights flown...hardly Big Data and deep analytics is it?

To solve this we really do need the return of the information scientist - someone who understands data, information and knowledge, who understands taxonomies, ontologies and the mathematical underpinnings of these. He or she also needs to know how to implement these, how best to organise a database, a data structure whether it be physically implemented in a triple store, relational database, flat log file etc...

For many this then rears the spectre of data governance and huge data management processes that slow the business down - you're not being agile are you then? Big Process is often the result of not understanding or not wanting to understand how to do things. If you have a number of apps and services wouldn't it be better to harmonise the information collection and ensure that the schemata and underlying semantics are consistent over all your collected data? Surely spending less time on data quality and consistency because quality and consistency are already inherent in your data is a much better use of your resources.

So, if you're having arguments over which NoSQL database to use, or whether triple stores and graph databases have the performance you require or even if Semantic Web is just an academic construct, then you're certainly not doing Big Data and the knowledge you will gain from the data you are capturing will only be supporting your business in the most ephemeral of ways.

DNT Dies...

It seems like the W3C's Do Not Track efforts have more lives and deaths than Schrödinger's cat and a recently bought Norwegian Blue parrot. Now after the previous effort of opening the cat's box and finding it alive, or at least, just merely sleeping, or maybe dumping some bird seed inside and hoping for the best, DNT might actually be dead - or at least either the vial of poison has been observed to break, or John Cleese just beat it over a shop counter.

So while the Digital Advertising Alliance exits the DNT group stating a number of reasons including ill-defined scope, no definition of tracking etc, we're back to the same place and that is we have once again failed to define what privacy actually means.

In the Do Not Track and Beyond (W3C Workshop) held back in 2012 I wrote the following in my paper:

We still do not have a good theory of privacy or even common terminological framework that unifies the engineers, scientists, mathematicians, lawyers and consumer advocates - let alone the end user - yet.

Failure to define what DNT is (or was), the semantics of tracking and the very definition of what privacy is in this context was probably the major factor in the failure of DNT to progress and bring disparate groups with differing notions of privacy successfully together.

Indeed the above is just one symptom of the malaise that is affecting privacy and also in a wider scheme the very ideas of data processing and Big Data. We are not spending any time thinking about the semantics or meaning of these. Within the privacy community even trying to start a discussion on the semantics of privacy is fraught with difficulties that only Machiavelli and Kafka could have dreamt about.

From an earlier article entitled on the Naivety of Privacy I wrote:

I think we're missing the point what privacy really is and certainly we have little idea at this time how to effectively build information systems with inherent privacy [3] as a property of those systems. I have one initial conclusion:

WE HAVE NO UNDERLYING THEORY OF PRIVACY

We have no common definitions, common language, common semantics nor mappings between our individual worlds: legal, advocacy and engineering. Worse, in each of these worlds terminology and semantics are not always so well internally defined.

Maybe, the current difficulties of the DNT group will force us to reassess what we mean by "privacy" and "tracking"?

The Art of Writing Good Documentation: Teach, Don't Tell

Just come across this posting on how to write documentation, or more specifically, how to write GOOD documentation. It is by Steve Losh and called Teach, Don't Tell.

For example:

If you use many open source libraries you’ve undoubtedly encountered some whose README says something like “read the source”. Every time I see one, I die a little bit inside.

Source code is not documentation. Can you learn to be a guitarist by simply listening to a piece of music intently? Can you become a painter by visiting a lot of museums? Of course not!

This is one of the main reasons why we gave up on brilliant languages, frameworks etc- for example, this is the main reason why we had to give up on Opa sadly.

This is always the case when the experts in a language, framework, library etc are those who are developing the langauge, framework or library. They are so consumed by developing they forget the people who want to develop with their creation.

Teach the developers how to do something, don't fob the developer off with function signatures, obscure examples etc. If a developer is asking naive and "stupid" questions then there's probably a damned good reason why.

Maybe reading Zen and the Art of Motorcycle Maintenance should be compulsory for all?

Fail fast, fail often, fail always

Apparently failing fast and failing often will lead to an eventual success, assuming of course that you are able to give the project or whatever it is time to develop and mature to a sufficient degree so that "failure" can be meaningfully assessed.

Failure does not necessarily mean that the project will not be a success, but rather that the technology, knowledge and product doesn't necessarily satisfy the current business goals. This is quite different from something that is a failure.

The mentality that everything should be an immediate success and that the smallest flaw is evidence of failure is killing not just research and innovation but the whole research and development process.

Furthermore the tendency to rapidly kill projects perceived to be failures without understanding what can be learned from those project and applied to bolster the next or parallel projects will eventually kill your business.

Innovation only comes from allowing scientists and engineers time to develop their ideas and not by rapidly killing those ideas before enough learning can be gained from them to benefit the overall business.

Indeed, if you do want to fail fast and fail often, ensure that you are learning from your mistakes and capitalising on the technologies and knowledge gained so that the fail often doesn't become fail always.

Wikileaks, NSA and a crytpographic cat and mouse game

So we know that certain government agencies are collecting data and we also know that Wikileaks has at least 400Gb of encrypted data (aka. "insurance") now in the public domain.

It is encrypted and no-one knows what's there...right?

Schneier posted on his blog an article about the upcomming cryptopocalypse about an presentation from the Black Hat conference entitled The Factoring Dead:Preparing for the Cryptopocalypse.

Given that we are fairly confident that the NSA has an enormous amount of computing power available to it, but even then probably not enough for a brute force attack on the Wikileak's encrypted data by a few orders of magnitude, they do have some pretty competent mathematicians working in that field and a vast amount of data including a lot of unencrypted originals.

Now, as mathematics starts chipping away as the encryption algorithms themselves the amount of computation and time to decrypt without a key comes dramatically down (it might just take a few billions of years now).

What helps in these scenarios is if you have some knowledge about what the encrypted data contains and its encryption method. If the Wikileaks file contains what it is supposed to contain (whatever that is, but it probably implicates the USA) then we have a head-start towards the decryption. This isn't too far from what Alan Turing and the Bletchley code breakers did.

Furthermore, rarely in the past have we have huge amounts of data to work with...encryption algorithms generate data with extremely high entropy. The point now is that is there enough entropy in 400Gb of encrypted data? Do patterns start emerging at some point? Admittedly even 400Gb is not considered a large amount of data these days.

Furthermore the encryption algorithm used is one approved by the NSA. What if the NSA start matching already encrypted documents they hold against the Wikileaks data? Is there a weakness in the keys? Are there patterns in the encrypted data that are independent of the keys being used? Then there's the interesting idea of a blackdoor in the algorithm too.

So, (tin foil hats on please), one of the things that Wikileaks is then looking forward is accidental disclosure of some information in that release. This might suggest that either NSA has succesfulyl decrypted the data. I'm sure that the NSA aren't that stupid to tell us that but governments are large things and even despite apparently good security practices breaches happen either deliberately or accidentally.

From a game theoretic point of view the best I can come up with at the moment is that any release of information by the NSA will reveal subtle hints about what they can and can not do with regards to their knowledge of encryption algorithms.

Shares in popcorn and publishers of books on game theory just went through the roof. Ultimately it might just be that physics, mathematics and plain old information theory has the last laugh...

DNT lives!

The W3C's DNT proposal continues to march on. Thanks to an anonymous colleague for the link, his DNT value is undefined so I assume it to be "1".

From a technical viewpoint, DNT is quite simple in that adding a line to the HTTP headers is an instruction to any downstream carrier or user of that information that you do not wish to be tracked. The definition of what is means to be tracked and how this is enforced is of course somewhat hard to ascertain, but:

DNT: 0              // you can track me
DNT: 1              // don't track me

This got me wondering, given recent revelations, maybe we should add extra parameters or extend DNT to take into account other organisations who might be viewing our data?

I propose a 2-bit value:

DNT: 00     // please track me and supply me with obscure, irrelevant advertisements etc.
DNT: 01     // DNT as currently defined by W3C (meaning advertisers)
DNT: 10     // NSA, GCHQ etc are not allowed to track me, but advertisers can
DNT: 11     // no-one, not even government agencies or advertisers may track me.

Flying planes, Surgery and Privacy - the presentation

Here's a version of a talk I gave with a colleague on our experiences in using checklists in our daily work auditing systems for privacy compliance. To me the surgical checklist is probably the most apt for the types of work we perform during a privacy audit. Then privacy steps beyond the bounds of just "privacy" to becoming a more information safety issue:


 

Flying Planes, Surgery and Privacy (external version) from Ian Oliver

Facebook Postings and Hair Pins

Two things I noticed today: firstly a posting appeared on my Facebook wall linking to an article from the Finnish newspaper Iltalehti about 'hair pins' and their correct usage - more about that later - and of course people posted replies. However I was wondering how many noticed the small grey text underneath:

What might be an innocent (or not) reply could potentially go not just to your friends, and their friends, but also to the entire readership of Iltalehti.

We can get into an interesting argument about whose fault this this and whether the person posting should have read that comment about the potential readership, however discussing fault and appointing blame isn't too helpful as we've discussed earlier in an article about understanding accidents.

None of the 3 parties involved here: Facebook, Iltalehti and the writer of the comment are entirely to blame nor entirely innocent, but just like the pilot who let his airspeed decay and crashed the aircraft, we have an overall education and situation awareness problem that is going to be very difficult to change.

I'm not even advocating not posting to Facebook, but just be more aware of where comments are going to end up in addition to what you think your Facebook privacy settings are.

Note for non-Finnish speakers: apparently hair pins are being used the wrong way round...you can put the above article's URL into Bing or Google to translate.

Aside: first time I used Bing translate, it worked perfectly....though it did translate Finnish to...Finnish....

Secondly, regarding the above article on hair pins...it reminded me of a quote from Douglas Adams' Hitchhikers Guide to the Galaxy:

The sign said: Hold stick near centre of its length. Moisten pointed end in mouth. Insert in tooth space, blunt end next to gum. Use gentle in-out motion. 

"It seemed to me," said Wonko the Sane, "that any civilization that had so far lost its head as to need to include a set of detailed instructions for use in a packet of toothpicks, was no longer a civilization in which I could live and stay sane."

Understanding Software Engineering Accidents

The cause of 99.999...% of accidents is easy to ascertain, quite simply it is the pilot's/driver's fault. In the cases of two recent aviation and railway accidents (Asiana 214 and the Galicia's train crash) these were caused by the pilot and driver respectively....?

Finding the causes of an accident don't actually involve finding who is to blame but rather the whole context of what led up to the point where an accident was inevitable. And then even after that exploring what actually occurred during and after the accident.

The reasoning here is that if we concentrate on who to blame then we will miss all the circumstances that allowed that accident to occur in the first place. For example., in the case of the Galician train crash it has been ascertained that the driver was speeding, on the phone and had a history of breaking the rules. However this misses the more subtle questions of why did the train derail, why could the driver break the speed limit, what was the reasoning of the phone call, why did the carriages fail catastrophically after the derailment, did the safety systems work sufficiently, what state was the signalling in, did the driver override systems, was the driver sufficiently trained etc etc etc.

In other words, the whole context of the systems and organisation needs to be taken into account before appointing final blame; and even then very, very rarely is it single point of failure: the Swiss Cheese Model.

If we apply this model to software engineering and specifically accidents such as hacking and data breaches we become very aware of how many holes in our computing Swiss cheese we have.

Take a simple data breach where "hackers" have accessed a database on some server via an SQL injection via some web pages. If we apply our earlier thinking, it is obviously the fault of the 'stupid' system administrators who can't secure a system and the 'stupid' software engineers who can't write good code. Hindsight is great here isn't it?

To answer 'who to blame?' or better still 'why did things go wrong and how can we prevent this in the future?' we need to put ourselves, as other accident investigators do, in the position of those software engineers, system administrators, architects, hackers and managers AT THE POINT IN TIME WHERE THEY ACTED, and NEVER in hindsight.

Why did the trained, intelligent software engineer write code susceptible to SQLi in the first place?

Maybe they were under time pressure, no proper, formal design documentation, coding standards, never trained to spot those kinds of errors? Actually just stating that we have a failure of discipline is already pointing to wholesale failures across our whole organisation rather than just one individual.

Even if in the above case this was malice by the programmer, then why didn't the testing pick this up? Why was the code released without testing or checking? Now we have a failure elsewhere too.

Were there no procedures for this? Was the code signed-off by management with the risk in place? Now the net widens further across the organisation

and so on...

In aviation there is a term to explain most accidents: loss of situational awareness, which when explored invariably ends up with a multitude of 'wrong' choices being made over a longer period of time rather than just at those few critical minutes or hours in the cockpit.

I'm of the opinion that in software engineering that we almost always operate in a mode where we have no or little situational awareness. Our systems are complex, we lack formal models of our systems that clearly and concisely explain how the system works; indeed one of the maxims used by some practitioners of agile methods actively eschews the use of formality and modelling tools. Coupled with tight deadlines, a code-is-king mentality and rapidly and inconsistently changing requirements we have a fantastic recipe for disaster.

Bringing this back to an aviation analogy again, consider Turkish Airlines Flight 1951 which crashed as Schiphol in 2009. It was initially easy to blame the pilots for allowíng the plane to stall on final approach, but the whole accident investigation revealed deficiencies in the training, the approach procedures of Schiphol, a non-fault tolerant autothrottle and radar combination, a massively high workload situation for the pilots and ultimately a fault which manifested itself in precisely the behaviour that the pilots were requiring and expecting on their approach, that is the aircraft was losing speed.

As an exercise, how does the above accident map to what we experience every day in software engineering? Given high workloads, changing requirements, inconsistent planning and deadlines to get something (anything!) out that sort of works and we start getting answers to why intelligent administrators and programmers make mistakes.

2001 again...

James Maynard Gelinas' blogpost on "Underground Research Initiative" entitled
2001: A Space Odyssey - Discerning Themes through Score and Imagery is a 22,000 word essay about the film 2001 and one certainly as worth reading as watching Kubrik's masterpiece.

In the 50 years since its release, while special effects have improved, the sheer grandeur of the film from many angles, especially that of science, has not diminished.

I can't say anything more other than what is already written in Gelinas' article, maybe than we got Facebook and Twitter instead, which given the effect of the Monolith in later books in Clarke's series might have been a good thing (though it seems that while 3001 told us how to destroy a Monolith, another movie used the idea - and a Macintosh - aliens (and humans too), beware of iPods).

OK, if I do add something, in 2001 you're probably going to see the best portrayal of a computer in any film (including ones using Macintoshes).

Maybe I should lobby Finnkino to show it on their number one screen in Helsinki....

Children of the Magenta

Children of the Magenta is a term used to describe pilots overly reliant on automation rather than trusting their own skills in flying an aircraft. This video from an American Airlines training session, where the term "children of the magenta" was coined - due to the colour of the flight director bug or icon - is essential viewing not just for pilots but for anyone involved in driving, managing or building systems of any kind:

In particular I want to present the bullet points from one slide and then address another aspect regarding levels of automation:

Automation Dependency

• The pilot flying should remain as 'one' with the aircraft in any low altitude maneuvering environment with the autopilot engaged
• To maintain situational awareness of both aircraft performance and flight path.
• The autopilot and autothrottles have limitations which affect performance
• Crews are returning to the autopilot in an attempt to resolve a deteriorating situation
• Autopilot and Autothrottles, however good, cannot recover the aircraft from a critical flight attitude

If we consider what we do in software engineering, and especially in areas where there are strong privacy and security aspects such as any form of web and database development, do we end up relying upon our skills and technique or do we blindly follow process and procedure? Do we blindly follow waterfall or agile processes to their logical conclusion and attempt to beat that deadline and deliver something knowing that the quality of the delivered product is deteriorating. Do we allow our processes to become our overly trusted autopilots?

Finally three levels of automation were presented: low, medium, high. The lowest corresponds to hand flying, the medium to using the autopilot to guide the plane and the highest using the flight management computer to run everything.

I want to discuss these in a later post, but for now I'll leave it as an exercise to map these level to how we act in a given software engineer project.

Probably the biggest takeaway from this lesson in aviation when applied to any aspect of software engineering is:

If you are losing control of any aspect of project then the process won't correct it; only you as a software engineer will - if you take control.

And that applies to any aspect of the system: functionality, performance, security, privacy etc etc etc...

Big Metadata

One of the issues I see with auditing systems for privacy compliance is actually understanding what data they are holding. Often it is the case that the project teams themselves don't understand their databases and log files sufficiently. Worse is that misinterpretation of the NoSQL and BigData approaches have left us in a situation where schemata can be forgotten - or at least defined implicitly at run-time. The dogma is that relational databases have failed and all this "non-agile", predefined, waterfall, defined schemata stuff is a major part of this.

Losing all this information about types and semantics is a huge problem because no longer can be fully sure of the consistency and integrity of the data and the relationships of that data to other objects and structures.

We are also then losing the opportunity to add additional information in the form of aspects to the data, for example, security classifications, broad usage classifications, and so on. This leads to embedding much of the information about the data statically into the algorithms that operation over that data; which in turn hides the meaning of the data away from the data itself.

I think this article entitled "Big Data success needs Big Metadata" by Simon James Gratton of CapGemini sums it quite quite well: forgetting about the meaning of data will seriously compromise our ability to understand, use and integrate the data in the first place!

To achieve this, good old fashioned data classification and cataloguing is required. Ironically this is exactly the stuff that database developers used to do before the onset of the trend to making everything schema-free.

Together with suitably defined aspects and ontologies that describe information (meta-information?) in much the same way as the OMG's MOF with additional structure and semantics we already have much, if not all, of the required infrastructure.

Then the process side of things needs to ensure that development of systems, services, analytics and applications integrates with this - in that whatever data they store (locally or in the cloud) gets recorded (and updated!). That's probably the hardest part!

See also:

• Metadata helps make data discoverable and accessible. Irfan Khan. IT World.  August 02, 2012 
• Metadata, Big Data, and the Financial Sector  Angela Guess on April 26, 2012. SemanticWeb.Com
• The MetaData Universe. Mervin Mooi. Web Business Intelligence. Feb 8, 2013

Systems Safety - Defining Moments

As I've been concentrating on "safety improvements", or at least techniques for the improvement of system I've tended to concentrate on four areas:

• Aviation
• Industrial
• Medical (specifically surgical)
• Software Engineering (specifically information privacy)

The parallels between theses areas when it comes to what each define as "safety" and the techniques should be obvious. However the question remains what actually triggered each of these respective areas to take a more inherent safety approach and then what will trigger a similar approach with regards to information safety?

 

Above Diagram Key: Y-Axis: relative degree of safety embedded into that discipline, X-Axis, year of time since seminal incident.

Aviation safety's seminal moment was the 1935 crash of a Boeing Model 299 aircraft during a presentation flight. Instead of blaming the pilots, effort was made to understand the causes of the accident and develop techniques to help prevent similar accidents in the future.

For industrial safety the seminal moment was the 1974 Flixborough Disaster in the UK. This resulted in work on the design of industrial plants and the development of the notion of "inherent safety".

Surgical safety has quite a long tradition especially with the development of anaesthetic safety from the 1960s and the introduction of a proper systems approach. However anesthetists seem not to feature prominently as surgeons and doctors so the fame would probably go to Peter Pronovost et.al. for the Central Line checklist. This was probably one of the major contributors to the WHO Surgical Safety Checklist discussed in detail in Atul Gawande's book The Checklist Manifesto which brings together much of the above incidents. 

If you're still in doubt maybe Atul Gawande's article in the New Yorker magazine entitled The Checklist: If something so simple can transform intensive care, what else can it do? (Dec 10, 2007) might help.

Getting back to the crux of this article, what is the incident that will cause the wholesale change in attitudes and techniques to software engineering that instills such a sense of discipline that we can eradicate errors to such a degree that we could compare ourselves favourably with other disciplines?

The increasingly frequent hacking and information leaks? The NSA wiretapping and mass surveillance? Facebook and Google's privacy policies? None of these have had any lasting effect upon the very core of software engineering if any at all. Which either means that we place such low value on the safety of our information or that the economics of software are so badly formulated in society that the catastrophe would have to be so huge that it would have to cause societal change?

Interestingly, in software engineering and computer science we're certainly not short on techniques for improving the quality and reliability of the systems we're developing: formal methods (eg: Alloy, B, Z, VDM etc), proof, simulation, testing, modelling (in general). What we probably lack is the simplicity of a checklist to guide us through the morass of problems we encounter. In this last respect, this is why I think we're more like surgeons that modern day aviators; or, maybe some of us are like the investigators to the 1935 Boeing crash and other aviation heroes learning their trade?

Asiana Flight 214 and Lessons for Software Engineering

Tony Kern's book Flight Discipline [1] discusses the notion of discipline related to flight safety through his experience flying, training and by example - almost invariably by way of a crash.

It is testament to the aviation industry overall how safe flying has become through the application of safety procedures and techniques, and also the very real need for pilots to be extremely disciplined in their work. The same concepts are also being applied in industrial and medical environments too as described earlier in this blog.

This week however has seen an accident at San Francisco airport involving Asiana Flight 214. The causes (note: plural!) are being investigated by the NTSB and I'm not in any position to appropriate blame, but I do think this will become quite a seminal accident in terms of what is found.

At the time of writing it is known that the aircraft was believe to be in a thrust managed mode which should have kept its speed and that the crew somehow allowed its speed to decay and mismanaged the final stage of landing. There are another set of possibly contributing factors such as CRM, crew fatigue, experience, the ILS being out of operation, a visual approach etc. This of course is compounded by the fact that hundreds of other aircraft have landed successfully at SFO in recent weeks under similar conditions.

Whether or not this is the fault of the pilots, a fault of the automatics on the aircraft or a combination of a host of those and other factors is to be ascertained. One thing that seems to be sure is that the pilots failed to monitor a critical aspect of the aircraft's situation. In Kern's terms the discipline of the pilots to maintain proper checks has failed. This will be an interesting case to follow, possible a seminal case with respect to discipline.

In terms of software engineering or specifically some aspects of developing with security in mind, I've been discussing with a colleague about SQL injection attacks and that these are remarkably easy to detect and defend against - though also remarkably easy to write. Given this, we still seem to be detecting vulnerable code remarkably often.

So this leads to the question why are we still writing code vulnerable to SQL injection if it is so easy to detect?

King puts forward four reasons for breakdown in discipline, which we can put into our context:

1.    Programmers don’t know the procedures for developing and testing code that is susceptible to SQL injection.
2.    Programmers don’t know the techniques to avoid writing code vunerable to SQL injection.
3.    Programmers forget either of points 1 and 2.
4.    Programmers willfully decide to ignore points 1 and 2.

Most programmers do know about the errors and pitfalls of particular implementation technologies, so the failure in discipline does not necessary lie mainly there. Even if it does, this is corrected simply by training.

Regrettably one of the excuses is always that the programmers are being “agile”: up against deadlines, constantly and inconsistently changing requirements, no or poor specifications, lack of emphasis on design and thought, and poor organisational communication. Errors can be fixed in a later sprint (if remembered and the if the fix is implemented). This then ends up with vulnerabilities such as SQL injection being forgotten by the programmers.

Willful ignorance of vulnerabilities follows from the previous point. As vulnerabilities are luckily not often exploited at run-time and if in place detection methods will probably catch these errors, then the programmers can often get away with vulnerable code. Furthermore, responsibility for the code nor the maintenance does not fall often with the original programmers or software engineering team.

Could it be that in software engineering we still have the hero mentality? Ironically it was many of the early 'hero' pilots who actively developed the procedures and techniques for safe flying.

Unlike software engineering the economics of bad flying proved to be a powerful incentive for actively pursuing good flying techniques and ultimately very sophisticated safety programmes. Maybe the only way of solving the discipline problem in software engineering is to radically alter the economic argument.

The economics of flying badly tends to end up with deaths and often the pilot’s own life (it is probably even worse if he/she survives) – the direct and collateral costs here are easily calculable. The economics of security errors in programming aren’t well understood nor calculated.

If a SQL injection releases one million records of personally identifiable information, why should the programmer care especially when there are little or no consequences for the programmer, the software engineering team, the managers and the business overall?

I don't suggest that we start killing programmers for errors, but maybe alter the way we prize teams for getting software into production. The fewer changes and bug fixes over the first year of a product's life = bigger bonus; inverting the bonus for bug fixing? Should we be more like hardware engineers and increase the costs of compilation of code such that a software engineering team gets only 1 chance per day to compile their code with penalties for compilation errors?

To finalise, maybe Tony Kern talking about discipline and what makes a great pilot - now apply this to software engineering, privacy, security etc etc:

References:

[1] Tony Kern (1998) Flight Discipline. McGraw Hill 0-07-034371-3

Facebook and Twitter....instead....

It was one of those funnily serendipitous things that connect everything together...first of all it started "playing" with XPlane 10....a damned good (understatement!) simulator and I suppose the next best thing to actually owning your own aircraft or holding a pilots license...anyway, flying around with some VOR or VOR navigation I ended up in Shannon --- and yes, I did land the plane.

Then I started wondering what flies out of Shannon now - the Shannon Stop-Over has been gone a very long time - and so I ended up looking for the Shannon departures board. Noticing the BA001 and 003 flights there - London City to JFK via Shannon on the outbound leg on an Airbus A318, I was wondering what that service was like.

Read a few passenger reports and noticed a comment about it being like the Pan Am Lunar Shuttle inside - certainly the photos of the all business-class seating look like it.

Which brings me to a clip of the said Lunar Shuttle from Kubrik's 2001:


Which then got me thinking:

1. The Blue Danube or more correctly An der schönen blauen Donau is a beautiful piece of music perfectly suited to a space docking
2. Kubrik was a genius
3. A.C. Clarke's 2001 is also a piece of genius
4. We had vision in the 1960's of a future where space travel and space-stations might be common place
5. We got Twitter and Facebook instead....

 ....which allows me to write about what we could have had instead of having it....