DSummit, Stockholm, May 2016

One for the CEOs, CIOs and CxOs of the world. This year DSummit is in Stockholm on 26th-27th May and has an impressive array of speakers and a strong focus on #privacy engineering!

"Disruptology is the art and science behind disruption. We study disruption and its impact on business and society. With a network of change makers, technology moguls and innovation evangelists we assist companies of all sizes with guidance, advisory and resources to become true disruptors. As an non profit academic institution and research foundation, Disruptology is a pioneer of new and disruptive business models, such as the F2W free-to-win model. With a vast network of industry professionals on call, we are able to inject new ways of thinking, working and playing into the DNA of companies throughout the world."

Tickets can be bought here: https://www.tickettailor.com/checkout/view-event/id/38963/chk/1862/ref/ian-oliver

And further details of the event here: http://www.dsummit.net/

Loyalty Cards

Tangentially related to privacy which I'll discuss later regarding information asymmetry, but first I wanted to comment on this article that appeared on LinkedIn: Business Travelers Are Saying "Buh-bye" to Loyalty Programs - Here's Why by Christopher Elliott, a "Reader advocate" for National Geographic Traveler

"After years of putting up with blackout dates, broken promises and bait-and-switch games, American travellers — particularly air travellers — are saying “Enough!”
They’re refusing to play the loyalty-program game, jettisoning blind brand allegiance in favor of a more pragmatic view of travel. Price and convenience are trumping mindless devotion to an airline, a car rental company or a hotel."

This applies equally to European travellers to I suspect. But let's look at the price-quality-customer service dimensions. I did as the article states a few years ago and switched allegiance from one to another, despite accumulating a sizeable amount of points and occasionally using them to upgrade it actually became too difficult to actually use those points.

I remember a few cases clearly. The first was trying to enter a business lounge at Helsinki Airport - according to the "loyalty" scheme terms and conditions I had enough flights to qualify for an upgrade to the next tier, thus allowing me access to the business lounge. Except that there was a mistranslation between the English translation and the "correct" terms and conditions. The airline customer service basically stated it wasn't their fault. Given that the lounge at the time was empty, but staffed by no less than three "customer representatives" make it all a little too surreal.

The second was at Heathrow when lounge entry was not just dependent on the card but on the airline that issued to card, despite both airlines being part of the same alliance. It worked like this: if you gained the second tier on airline A you could enter business lounges run by the alliance that A was a member of, except, if the lounges were at an airport where airline B, also of the same alliance, was the major carrier.

The third was when trying to book a flight with air miles. For my selected flight I had enough miles and successfully made the booking until it came to select my seats, where upon the whole process failed and I was unceremoniously ditched out of the booking process with a terse error about the flight no longer being available to points holders. It turns out this was an artefact of the internal IT systems and actually you had to call the customers service department to book the flights - after a long hold where you were constantly told that all of the customer service representatives were busy, but you really are valued by the company you're trying to spend you money with.

After a few flights across the Atlantic I switched to Lufthansa - Frankfurt and Munich are clean, efficient and as pleasant as airports go. Lufthansa may not have good economy seating on long haul but the crews are some of the most professional (and smiling!) I've had the pleasure to be on an aircraft with.

I actually have no idea actually how to redeem the points I've collected with Lufthansa and SAS - they'll probably expire or have expired without my knowledge without me ever receiving any bonus from my loyalty as a customer.

A few years ago a colleague of mine called a major Nordic airline in response to a request for customer feedback. It turns out that they couldn't care why he stopped flying with them. Surely with all the data analytics being applied to our data collected, the airline never stopped for a moment to question why customers suddenly stopped flying with them. 

Now due to primarily economic reasons, I fly with whoever gives me the best deal. Given that the terms and conditions are pretty much the same regardless whether you fly a national carrier or a "cheap" airline, price will always win. 

Given that many established, major airlines decided to compete with the cheap carriers by matching their standards and terms, it is no wonder that the loyalty of the customer to the airline has all but disappeared?

Then, we come to credit cards and store cards...will these go the same way as air miles as customers realise that the deal they get is barely worth the loyalty?

But as a final statement, given that this is all about customer loyalty - has anyone ever been contacted by an airline, shop or other about why they stopped using those products offered?

References:

"The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2001: Information for the Public", press release from the Royal Swedish Academy of Sciences, Nobel Foundation, nobelprize.org, October 2001

Understanding Software Engineering Accidents

The cause of 99.999...% of accidents is easy to ascertain, quite simply it is the pilot's/driver's fault. In the cases of two recent aviation and railway accidents (Asiana 214 and the Galicia's train crash) these were caused by the pilot and driver respectively....?

Finding the causes of an accident don't actually involve finding who is to blame but rather the whole context of what led up to the point where an accident was inevitable. And then even after that exploring what actually occurred during and after the accident.

The reasoning here is that if we concentrate on who to blame then we will miss all the circumstances that allowed that accident to occur in the first place. For example., in the case of the Galician train crash it has been ascertained that the driver was speeding, on the phone and had a history of breaking the rules. However this misses the more subtle questions of why did the train derail, why could the driver break the speed limit, what was the reasoning of the phone call, why did the carriages fail catastrophically after the derailment, did the safety systems work sufficiently, what state was the signalling in, did the driver override systems, was the driver sufficiently trained etc etc etc.

In other words, the whole context of the systems and organisation needs to be taken into account before appointing final blame; and even then very, very rarely is it single point of failure: the Swiss Cheese Model.

If we apply this model to software engineering and specifically accidents such as hacking and data breaches we become very aware of how many holes in our computing Swiss cheese we have.

Take a simple data breach where "hackers" have accessed a database on some server via an SQL injection via some web pages. If we apply our earlier thinking, it is obviously the fault of the 'stupid' system administrators who can't secure a system and the 'stupid' software engineers who can't write good code. Hindsight is great here isn't it?

To answer 'who to blame?' or better still 'why did things go wrong and how can we prevent this in the future?' we need to put ourselves, as other accident investigators do, in the position of those software engineers, system administrators, architects, hackers and managers AT THE POINT IN TIME WHERE THEY ACTED, and NEVER in hindsight.

Why did the trained, intelligent software engineer write code susceptible to SQLi in the first place?

Maybe they were under time pressure, no proper, formal design documentation, coding standards, never trained to spot those kinds of errors? Actually just stating that we have a failure of discipline is already pointing to wholesale failures across our whole organisation rather than just one individual.

Even if in the above case this was malice by the programmer, then why didn't the testing pick this up? Why was the code released without testing or checking? Now we have a failure elsewhere too.

Were there no procedures for this? Was the code signed-off by management with the risk in place? Now the net widens further across the organisation

and so on...

In aviation there is a term to explain most accidents: loss of situational awareness, which when explored invariably ends up with a multitude of 'wrong' choices being made over a longer period of time rather than just at those few critical minutes or hours in the cockpit.

I'm of the opinion that in software engineering that we almost always operate in a mode where we have no or little situational awareness. Our systems are complex, we lack formal models of our systems that clearly and concisely explain how the system works; indeed one of the maxims used by some practitioners of agile methods actively eschews the use of formality and modelling tools. Coupled with tight deadlines, a code-is-king mentality and rapidly and inconsistently changing requirements we have a fantastic recipe for disaster.

Bringing this back to an aviation analogy again, consider Turkish Airlines Flight 1951 which crashed as Schiphol in 2009. It was initially easy to blame the pilots for allowíng the plane to stall on final approach, but the whole accident investigation revealed deficiencies in the training, the approach procedures of Schiphol, a non-fault tolerant autothrottle and radar combination, a massively high workload situation for the pilots and ultimately a fault which manifested itself in precisely the behaviour that the pilots were requiring and expecting on their approach, that is the aircraft was losing speed.

As an exercise, how does the above accident map to what we experience every day in software engineering? Given high workloads, changing requirements, inconsistent planning and deadlines to get something (anything!) out that sort of works and we start getting answers to why intelligent administrators and programmers make mistakes.

Systems Safety - Defining Moments

As I've been concentrating on "safety improvements", or at least techniques for the improvement of system I've tended to concentrate on four areas:

• Aviation
• Industrial
• Medical (specifically surgical)
• Software Engineering (specifically information privacy)

The parallels between theses areas when it comes to what each define as "safety" and the techniques should be obvious. However the question remains what actually triggered each of these respective areas to take a more inherent safety approach and then what will trigger a similar approach with regards to information safety?

 

Above Diagram Key: Y-Axis: relative degree of safety embedded into that discipline, X-Axis, year of time since seminal incident.

Aviation safety's seminal moment was the 1935 crash of a Boeing Model 299 aircraft during a presentation flight. Instead of blaming the pilots, effort was made to understand the causes of the accident and develop techniques to help prevent similar accidents in the future.

For industrial safety the seminal moment was the 1974 Flixborough Disaster in the UK. This resulted in work on the design of industrial plants and the development of the notion of "inherent safety".

Surgical safety has quite a long tradition especially with the development of anaesthetic safety from the 1960s and the introduction of a proper systems approach. However anesthetists seem not to feature prominently as surgeons and doctors so the fame would probably go to Peter Pronovost et.al. for the Central Line checklist. This was probably one of the major contributors to the WHO Surgical Safety Checklist discussed in detail in Atul Gawande's book The Checklist Manifesto which brings together much of the above incidents. 

If you're still in doubt maybe Atul Gawande's article in the New Yorker magazine entitled The Checklist: If something so simple can transform intensive care, what else can it do? (Dec 10, 2007) might help.

Getting back to the crux of this article, what is the incident that will cause the wholesale change in attitudes and techniques to software engineering that instills such a sense of discipline that we can eradicate errors to such a degree that we could compare ourselves favourably with other disciplines?

The increasingly frequent hacking and information leaks? The NSA wiretapping and mass surveillance? Facebook and Google's privacy policies? None of these have had any lasting effect upon the very core of software engineering if any at all. Which either means that we place such low value on the safety of our information or that the economics of software are so badly formulated in society that the catastrophe would have to be so huge that it would have to cause societal change?

Interestingly, in software engineering and computer science we're certainly not short on techniques for improving the quality and reliability of the systems we're developing: formal methods (eg: Alloy, B, Z, VDM etc), proof, simulation, testing, modelling (in general). What we probably lack is the simplicity of a checklist to guide us through the morass of problems we encounter. In this last respect, this is why I think we're more like surgeons that modern day aviators; or, maybe some of us are like the investigators to the 1935 Boeing crash and other aviation heroes learning their trade?

Asiana Flight 214 and Lessons for Software Engineering

Tony Kern's book Flight Discipline [1] discusses the notion of discipline related to flight safety through his experience flying, training and by example - almost invariably by way of a crash.

It is testament to the aviation industry overall how safe flying has become through the application of safety procedures and techniques, and also the very real need for pilots to be extremely disciplined in their work. The same concepts are also being applied in industrial and medical environments too as described earlier in this blog.

This week however has seen an accident at San Francisco airport involving Asiana Flight 214. The causes (note: plural!) are being investigated by the NTSB and I'm not in any position to appropriate blame, but I do think this will become quite a seminal accident in terms of what is found.

At the time of writing it is known that the aircraft was believe to be in a thrust managed mode which should have kept its speed and that the crew somehow allowed its speed to decay and mismanaged the final stage of landing. There are another set of possibly contributing factors such as CRM, crew fatigue, experience, the ILS being out of operation, a visual approach etc. This of course is compounded by the fact that hundreds of other aircraft have landed successfully at SFO in recent weeks under similar conditions.

Whether or not this is the fault of the pilots, a fault of the automatics on the aircraft or a combination of a host of those and other factors is to be ascertained. One thing that seems to be sure is that the pilots failed to monitor a critical aspect of the aircraft's situation. In Kern's terms the discipline of the pilots to maintain proper checks has failed. This will be an interesting case to follow, possible a seminal case with respect to discipline.

In terms of software engineering or specifically some aspects of developing with security in mind, I've been discussing with a colleague about SQL injection attacks and that these are remarkably easy to detect and defend against - though also remarkably easy to write. Given this, we still seem to be detecting vulnerable code remarkably often.

So this leads to the question why are we still writing code vulnerable to SQL injection if it is so easy to detect?

King puts forward four reasons for breakdown in discipline, which we can put into our context:

1.    Programmers don’t know the procedures for developing and testing code that is susceptible to SQL injection.
2.    Programmers don’t know the techniques to avoid writing code vunerable to SQL injection.
3.    Programmers forget either of points 1 and 2.
4.    Programmers willfully decide to ignore points 1 and 2.

Most programmers do know about the errors and pitfalls of particular implementation technologies, so the failure in discipline does not necessary lie mainly there. Even if it does, this is corrected simply by training.

Regrettably one of the excuses is always that the programmers are being “agile”: up against deadlines, constantly and inconsistently changing requirements, no or poor specifications, lack of emphasis on design and thought, and poor organisational communication. Errors can be fixed in a later sprint (if remembered and the if the fix is implemented). This then ends up with vulnerabilities such as SQL injection being forgotten by the programmers.

Willful ignorance of vulnerabilities follows from the previous point. As vulnerabilities are luckily not often exploited at run-time and if in place detection methods will probably catch these errors, then the programmers can often get away with vulnerable code. Furthermore, responsibility for the code nor the maintenance does not fall often with the original programmers or software engineering team.

Could it be that in software engineering we still have the hero mentality? Ironically it was many of the early 'hero' pilots who actively developed the procedures and techniques for safe flying.

Unlike software engineering the economics of bad flying proved to be a powerful incentive for actively pursuing good flying techniques and ultimately very sophisticated safety programmes. Maybe the only way of solving the discipline problem in software engineering is to radically alter the economic argument.

The economics of flying badly tends to end up with deaths and often the pilot’s own life (it is probably even worse if he/she survives) – the direct and collateral costs here are easily calculable. The economics of security errors in programming aren’t well understood nor calculated.

If a SQL injection releases one million records of personally identifiable information, why should the programmer care especially when there are little or no consequences for the programmer, the software engineering team, the managers and the business overall?

I don't suggest that we start killing programmers for errors, but maybe alter the way we prize teams for getting software into production. The fewer changes and bug fixes over the first year of a product's life = bigger bonus; inverting the bonus for bug fixing? Should we be more like hardware engineers and increase the costs of compilation of code such that a software engineering team gets only 1 chance per day to compile their code with penalties for compilation errors?

To finalise, maybe Tony Kern talking about discipline and what makes a great pilot - now apply this to software engineering, privacy, security etc etc:

References:

[1] Tony Kern (1998) Flight Discipline. McGraw Hill 0-07-034371-3

What is the economic component of privacy to the end-user?

Privacy is framed in terms of ethical, moral and economic arguments. Certainly most research supports that nearly everyone is worried about their on-line privacy, yet rarely if at all we've seen a company actually fail because of their attitude to privacy. Some companies have changed their ways admittedly, but the change has been minimal and after the initial fuss it is very much business as usual. Privacy seems not to be much of an ethical or moral issue to most people but rather an emotional one.

Given the above we don't see much of an economic argument either, at least not from the end-user side of things. The economic argument for using certain social networking services, sharing personal photographs, details etc seems to outweigh the potential disadvantages - which tend to be worries about to where data is being shared or sold, or who might be viewing that material.

Privacy is rarely framed in terms of an economic argument beyond the "data mart" idea but rather in emotional terms: nothing to hide, nothing to fear perhaps? Privacy is lost in the security and liberty arguments too being equated with concealment and freedom. Consider the argument put forward by US congressman Paul Rand [1] on privacy.

Laws are changing to emphasise the economic argument to the companies that provide internet services and there we do see a strong argument for a more robust privacy function. Whether we on the business side of things are addressing privacy in the right manner and in the same economic terms as our end-users is another matter altogether. For companies the economic argument for developing and adhering to an information safety (privacy) is very clear. We have a good economic argument for treating end-users' information with respect...whatever that is.

For the end-user the economic argument is unclear. Why should an end-user choose one service provider over another? What are the terms of this economic argument and have they even been defined at all?

References

[1]  Senator Rand Paul Talks Tech, Civil Liberties, and Keeping the Government Out of Your Email. By Spencer Ackerman 05.30.13 Wired

Some thoughts on Gamfication

It seems everyone is talking about gamification: the idea that any service can be turned into a game, reasoning that through reward and bargaining mechanisms a user will interact more with that given service for presumably greater rewards.

Although gamification seems to be the current zeitgeist, the idea is relatively old and the new part is this explicit in the customer-server interaction and inherent in the development process of applications and services. 

Many of the underlying concepts of gamification come from Game and Economic Theories such as those extensively researched and developed by Nash, von Neumann and. It is the principle behind the usage of store cards, air miles, nearly every loyalty scheme and most recently (in internet terms) the idea behind the “free” service. 

Despite gamification now coming to the fore, gamification is already relatively well established in many areas including privacy, although not as explicitly as currently being suggested.  For example you get your social networking (or blogging provision!) for free by giving up your privacy – you get a free service (and advertisements) and the service provider gets your behavioural profile. 

We can simply demonstrate this aspect of gamification in privacy by constructing a small normal form model using Game Theory techniques. Consider the payoff  between a customer and some service – if the customer registers with that service then they obtain an enhanced service:

		Service
		Basic	Enhanced
	Anonymous	Random advertisments, No personalization, No identification of user, No targeting, No profiling	0
Customer	Pseudo-Anonymous	Random advertisements, Some session provisioning, Pseudo anonymous tracking Limited profiling – non traceable	0
	Registered/Identified	0	Targeted advertising Full user experience, Customisable, Profiled

The difficulty here is assigning values to the sets of features available versus the data collection. In the above example disabling cookies has probably better payoff than leaving them enabled, primarily at the expense of the quality of data being collected by the service versus the consumers’ privacy. If we consider what the enhanced service might provide then the fact that the consumer is being profiled is outweighed by the advantages of the enhanced service. Certainly in the above, it is in the service’s interests to ensure that the benefit to the consumer outweighs any privacy (or other) concerns.

Payoff matrices for services such as Facebook, Google, Skype, Amazon etc, can be similarly constructed? It might just be that a well-defined payoff is what is contributing to those services’ popularity when put in the context of privacy.

Things get more interesting when services are combined. For example would letting Amazon provide Ikea with your purchase details being a step too far regarding personal privacy? Maybe Amazon should consider teaming up with Ikea… 10% discount vouchers on book cases for every 20 books bought…as long as you tell Amazon your Ikea Loyalty Card number. Just no recommendations for books on interior decorating please!

To further strengthen the gaming link, in the above the customer is also presented with a challenge in optimizing the reward – what is the cheapest way of getting a cheaper bookcase?

It might also just be that gamification becomes one of the drivers behind improving the quality of Big Data. The more “points” your score using a service, the better the quality of the profile and better service. The question here then becomes are we as individuals (or even as collectives) that interesting or for that matter, are we as consumer that discerning in the content from service providers and the back-and-forth bargaining? Do we care enough to be interested in playing the service improvement game?

Will gamification improve on the experience for the consumer and turn us all in to marketers of our own information?  Will recasting your service as a game lead to greater popularity, more consumers and success? 

What interests me, assuming the above answers are positive, is then how to do this through the embedding the economic principles of gamification inherently into the system design.

References

• The Gamification Wiki
• Gamification Books

Code is Law, Inherent Privacy and a Few Uncomfortable Issues

Lawrence Lessig stated that "code is law" - a maxim that above all should be the most critical in software engineering, especially when put in the context of implementing privacy and security.

I want to talk about some issues that worry me slightly (ok, a lot!). The first is that despite of policies, laws etc, the final implementation of anything related with privacy is in the code the programmers write. The second is that we are building our compliance programmes upon grand schemes and policies and paying piecemeal attention to the actual act of software engineering. The latter we attempt to wrap up in processes and "big ideas", for example, Privacy by Design.

Now before the PbD people get too upset, there's nothing wrong with stating and enumerating your principles, the Agile Manifesto is a great example of this, however there is no doubt that many implementation of agile are poor at best and grossly negligent and destructive at worst. The term used is "technical debt".

Aside: the best people I've seen conduct software development in an agile manner are formal methods people...I guess due to the discipline and training in the fundamentals they've received. This also applied to experienced architects, engineers and programmers for whom much of this formality is second nature.

Addressing the first point: no matter how many policies or great consumer advocates or promises you make, at the end of the day, privacy must be engineered into the architecture, design and code of your systems. It does not matter many powerpoint slides or policy documents or webpages your write, unless the programmers "get it", you can forget privacy, period!

Aside: Banning powerpoint may not be such a bad idea....

Herein lies a problem, the very nature of privacy in your systems means that it crosscuts every aspects of your design and ultimately your whole information strategy. Most of these things do not obviously manifest themselves in the design and code of your systems.

To solve this there must be a fundamental shift from the consumer advocacy-legal focus of privacy to a much deeper, technical or engineering, even scientific approach. This however does not just mean focusing on the design and code, though that is fundamental to the implementation, but to the whole stack of management and strategy from the highest directors to the programmers.

I've seen efforts in this direction but stop at the product management - "Hey, here are the privacy requirements - implement them!" ... which does feel good in that you are interacting, or believe that you are interacting, with the products you are producing but still not sufficiently with the people who really build these. Just producing requirements doesn't help: you need that interaction and communication right across the company.

Of course all of the above is extremely difficult and leads us to our next point which is how we build our compliance programmes in the first place. The simple question here is "are you fully inclusive?", meaning do you include programmers, architects (technical people with everyday experience) or is the programme run by non-technical, or formerly technical staff? Invariably it is the latter.

Compliance programmes must be inclusive otherwise the necessary inherency required to successfully and sufficiently implement the ideas and strategies of that programme will be lost - usually in a sea of powerpoint and policy documents.

Firstly in order to achieve inherent privacy (or security, or xyz) focus must lie on onboarding and educating the programmers, the designers and the architects and less focus on the management, prescription and consumer advocacy. Secondly, any compliance programme must be inclusive and understand the needs of the said technical staff. Thirdly, the engineering and technical staff are the most critical components in your organisation.

Compliance programmes are often measured on the amount of documentation produced (number of slides even?), however this ends up with a self feeding process where for the compliance programme to survive it needs to keep the fear of non-compliance at the fore. Read Jeff Jarvis' article on Privacy Inc.: Scare and Sell and then Jim Adler's talk about PII2012 on The Emergent Privacy-Industrial Complex and you get an idea of what is going wrong. Avoid at all costs creating a privacy priesthood in your compliance programmes.

Aside: This might be good old fashioned economics - if a compliance programme actually worked then there'd be no need for the programme in the end.

There are two interrelated caveats that also need to be discussed, the first of which is that any work in privacy will expose the flaws, holes and crosscutting issues across your products, development programmes and management and engineering skill bases. For example, a request to change a well crafted design to cope with some misunderstood ambiguity in privacy policy is going to end in tears for all concerned. It will demand of management, engineering and your compliance programme a much deeper [scientific] knowledge of what information your products are using, carrying, collecting and processing - to a degree uncommonly found in current practices.

The fundamental knowledge require to really appreciate information management and privacy is extensive and complex. Awareness courses are a start but I've seen precious few courses even attempting to cover the subject of privacy from a technical perspective.

Secondly, privacy will force you to examine your information strategy - or even create and information strategy - and ask very awkward and uncomfortable questions about your products and goals.

Finnish Customer Service

Customer service in Finland comes in two forms:  indifferent and diabolical, not so much "caveat emptor but "forsit est emptori", so it makes a change when you get customer service that can only be described as well above the call of duty.

I tried to buy a new jacket from Retikiaitta's web shop only to find a few problems with the availability and that the site would give different answers, either available or not, depending upon when the page was loaded. Anyway, I left a comment stating so in the vain hope that someone might do something.

About 2 hours later I get a call from the manger of the web shop apologising for the problem, followed by an interesting discussion on webpage caching, javascript, server-side caching, database issues etc. Finally he checked their central inventory, phoned Retkiaiita's store in Helsinki and reserved the jacket for me.


Customer service as its best. So a personal recommendation to use Retkiaiitta!!!

Now if only some other shops in Finland actually made the effort to attract and keep their customers - mentioning no names but two major DIY chains one of whom I had to use the Kuluttajavirasto to force them to admit their mistake and correct it and the other whom I found out later acted illegally according to Finnish Law. And then there's a major supermarket chain (there are only two in Finland) where two of their staff at a store in Helsinki acted in a manner which can only be described as racist.

Actually it amazes me often, that given the state of the current economy and just general business practice how anti-customer some businesses are and actually in some cases are openly hostile to taking my money.

Unintended Consequences in Academia

Found this via a posting on Facebook; a very interesting look at the unintended consequences of trying to measure the output of academics, and just like any other metric the issues that come up on how that particular metric gets misused. Sort of reminds me of the classic software engineering metric leading to bonus payments: bug fixing....

Perverse Incentives in Academia

 In particular I quite like these two:

Researchers rewarded for increased number of citations.

Researchers do work that is relevant and influential.

H-index obsession; list of references no longer included in page limit at many conferences.

Teachers rewarded for increased student test scores.

Improve teacher effectiveness.

Teaching to the tests; emphasis on short-term learning.

Of course, one issue here is that you are trying to measure some seriously intelligent people and here's a very good discussion on managing intelligent people by Scott Berkun - I'm not sure he has all the answers but just acknowledging this is a step to proper understanding.

Finally, a nod to that classic tome of unintended consequences (and the economic theory that drives this): Freakonomics by Levitt and Dubner.

The Elusive Big Idea

An excellent article from the New York Times on "The Elusive Big Idea"; the thesis that in today's society we just don't care about ideas anymore...

The Elusive Big Idea
By NEAL GABLER
Published: August 13, 2011

...

If our ideas seem smaller nowadays, it’s not because we are dumber than our forebears but because we just don’t care as much about ideas as they did. In effect, we are living in an increasingly post-idea world — a world in which big, thought-provoking ideas that can’t instantly be monetized are of so little intrinsic value that fewer people are generating them and fewer outlets are disseminating them, the Internet notwithstanding. Bold ideas are almost passé.

...

Post-Enlightenment refers to a style of thinking that no longer deploys the techniques of rational thought. Post-idea refers to thinking that is no longer done, regardless of the style.

The post-idea world has been a long time coming, and many factors have contributed to it. There is the retreat in universities from the real world, and an encouragement of and reward for the narrowest specialization rather than for daring — for tending potted plants rather than planting forests.

...

Scary thought indeed that "thinking is no longer done"; the implications of this will extend everywhere and affect every facet of our existence.

A further quote from the article:

...we get instant 140-character tweets about eating a sandwich or watching a TV show. While social networking may enlarge one’s circle and even introduce one to strangers, this is not the same thing as enlarging one’s intellectual universe.

A direct reference to Twitter and social media overall - the nature of communication is changing such that if you can't get your idea in 140 characters no-one is going to read it - assuming that anyone reads and thinks about what is actually contained in those 140 characters.

Is this some kind of existentialist crisis for ideas? And finally as a conclusion:

We have become information narcissists, so uninterested in anything outside ourselves and our friendship circles or in any tidbit we cannot share with those friends that if a Marx or a Nietzsche were suddenly to appear, blasting his ideas, no one would pay the slightest attention, certainly not the general media, which have learned to service our narcissism.

As Levitt and Dubner state in the Freakonomics series of books ( see: Superfreakonomics ) humans need incentive, and if ideas don't provide any incentive in either their thinking, development or execution to our narcissism then those ideas have no value. The problem here is rather self-referential in that narcissists aren't interested in anyone else's ideas - be honest who reads anyone's Twitter feeds or Facebook comments and partakes in deep rhetoric?