Perverse paradoxical feedback loop

I've always been fascinated with systems analysis, have picked up bits and pieces.
One key aspect of any self-regulating open system is the feedback loop - approval ratings for politicians, applause for actors, telemetry for rockets, whatever. Once you look for it, it's everywhere. You also tend to notice …
Systems tend to fuck up. Systems tend to do the opposite of what they're designed (or have evolved) to do.

The common thread I've noticed in a lot of this is what I call "the perverse paradoxical feedback loop."

Normally, positive feedback keeps you ("you" including humans, robots, natural systems, whatever) on track when you're achieving or getting closer to some goal; negative feedback zaps you when you're of course, not getting it. Sometimes it works the other way around - hence perverse. It's often predictable, but at the same time, also a case of systems doing exactly the opposite of what they're designed for - i.e., IT SHOULDN'T HAPPEN - hence, paradoxical.

Examples - 

Railroad fatalities in 1810 are X per number of miles traveled. Over the years, designers improve the safety level of trains, tracks, tunnels, grades, whatever. In 1830, the level of fatalities is still X. The reason: engineers compensated for the increased safety by taking more risks.

A guy and a chick are having an argument. He's irritated, is driving too fast, a little erratically. She says STOP DRIVING LIKE AN ASSHOLE. The result: he drives faster, more erratically, more like an asshole. The reason: responding to the negative feedback would mean admitting, in fact, he was driving like an asshole.

Many of the perverse results arise because humans are aware of feedback systems and work to circumvent them. Why study, when you can whine and cry until your teacher into gives you a good grade? Why put on a good play, when you can use advertising pressure to bully a newspaper into giving a good review? Why listen to hard facts now that you're a Hollywood STAR when you can surround yourself with sycophants? Why accept bad news when you can shoot the messenger?

Perverse results especially crop up when the feedback loop is digital (on-off) as opposed to analog (a series of gradual corrections). I.e.: zero tolerance. If making ANY MISTAKE means getting fired, I won't correct my mistakes, I will do my best to cover up my mistakes and put the blame on someone else - hence there's absolutely no feedback.

So: airplane pilots are supposed to have perfect vision. The minute your eyesight weakens, you're out. The result is not pilots with perfect vision but pilots doing their best to hide any defects - not wearing glasses, cheating on eyetests, taking exams on their own time. (Lasik surgery's probably made this a dated example.)

Broadly speaking, if you create a system where anything less than perfection is failure, you destroy any possibility of self-correction through feedback. The ideal is constant real-world feedback of every step of the process from idea to execution. Demming got into this in the industrial process. You can apply the same principle to marriages, relationships, philosophical thought, whatever.

But the only way it works is if it's safe to be open.

Hence: if we approach 9-11 as a "catastrophic failure of the intelligence community" with the intention that someone - better still, a whole line of someone's - must be crucified for it, a nice little Appian Way of crucified FBI, CIA, NSA and other governmental fuck-ups, because it HAS TO BE SOMEONE'S FAULT - the result will be that everyone involved right on down the chain will do their best to make everything about what they do opaque - to cover asses, shift blame, destroy, disappear or lose evidence, not put two and two together, to stonewall, obfuscate, lie or find sacrificial lamb if need be because IT'S YOUR CAREER. The result will be we'll learn very little or nothing, won't refine, fix or improve the process, won't learn any lessons, won't be ready next time.

The smart thing is to say what happened was, literally, psychotic and inconceivable - that it didn't occur to us because we are a healthy, open, non-psychotic democratic society - that they're the motherfuckers, not us - that the thing to do is dispassionately examine the chain of what happened and figure out what to do differently so it'll never happen again. But that can't happen if you're looking for scapegoats.

It has to be safe to be open.