Hal destroys Wall Street

Hal Quickly: automation is outstripping human common sense.

Content: Two cases are relevant. 1) Google's dissemination of old United Airlines news resulting in a precipitous UAL stock slide, and 2) exotic, computer-generated financial instruments leading Wall Street into fatal waters.

What happened? In the first case, a system unchecked by human beings made a mistake resulting in massive damage to an already fragile public company. But that's nothing compared to the wreckage caused by the inscrutable mortgage securities.

My father was a part-time banker who loved to extol the virtues of what he called "small town banking." It was all pretty simple: his bank took deposits and loaned that money to people whom the bank officer had gone to high school with. This was important, because bankers are not the highest IQ animals on the savanna -- they needed good doses of trust and information to sleep at night.

Step forward forty years to sub-prime mortgages. Using complex algorithms, Goldman, Lehman and others created securities that blended mortgages of divergent risk. These securities were supposedly safe, backed by another exotic instrument -- credit default swaps. The trust and information of the old days was replaced by computer models and arcane mathematics. The evil great-great computer grandchildren of Hal were in charge.

An east coast, mid-sized commercial bank has avoided this mess. How? In 2006, when the computer-concocted financial instruments were raging, his VPs said, "We have to get into this market." He said that he would consider, but first he demanded a two hour presentation on how the securities worked. He listened for two hours, then sat up in his chair and said, "We're not going to do this -- because I can't understand it."  That wasn't the case with Wachovia, Washington Mutual, Lehman and others -- they had their oxygen supplies cut by Hal (rent 2001: A Space Odyssey if you're under 40).

Automation and computers do much good in the world. But we have stepped over the line into a world where, without the check of human common sense and understanding, they can do great harm.

What it means one: Apply a simple rule: "If it doesn't make sense, it doesn't make sense." Don't believe that computer-driven models can reside beyond the realm of human understanding. Either grasp their import, or pull their plug.

What it means two: Risk assessment and management programs (perhaps within Sarbanes) should be placed on alert to identify the Hals in your company. Pinpoint the danger.

What it means three: If you are the leader, never be afraid to say Andy Grove's favorite business word: "No."

Let me know if you have any other examples of computing out of control...I'd love to get your thoughts.

Comments

re: Hal destroys Wall Street

This is an important point you raise here. Computers are not bad, but they can be used badly. In fact we could pinpoint this further to the use of spreadsheets. I have spent much time in my career following formulae from the output screen back to the source, and invariably there is an error, at worst, or at best an malformed assumption. Yet Banks and all kinds of companies live on spreadsheets.CEO's - learn spreadsheets is my suggestion, so you can understand what you are being sold from your finance dept, strategy dept etc etc.re other examples:- search engines - how do you know the defnition of what rises to the top?This has made me think of another nuance. Thats the difference between mathematics and algorithms. The former is precise. The latter is based on assumptions.I think the issue you raise is relative to algorithms, where risk models based on sets of assumptions led to conclusions that people took as 100%, but they were in fact +/- xx%

re: Hal destroys Wall Street

George, thank you so much for writing this blog. I have made this point so often in the last years and was always laughed away. Even Nassim Nicholas Taleb a former wall street trader, who wrote the New York Times bestseller 'The Black Swan' could not make people understand that mathematically calculating maximum future utility is pure guesswork. People loved 'Super Crunching' by Ian Ayres who claimed that 'anything can be calculated.' In my July 2007 book review on Amazon and my blog http://maxjpucher.wordpress.com/2007/07/24/book-review-super-crunching-b... tried to present the view that the credit crisis was caused by this dangerous path.I have spent many years researching the scientific findings on human decision making and why it works so well as it does with limited information. Read Damasio, Gigerenzer, Pinker and Minsky to learn that the human mind is perfectly adapted to this uncertain world. I have stopped investing on the stock market when I realized I could no longer follow the decision making. That was about ten years ago and it saved me a lot of trouble and worries. My research also led to different software. The User-Trained Agent of our Papyrus process management solution learns from this uncertain decision making of the human clerk without needing complex mathematical guesswork.The key question now is: Will we learn from this and apply it to IT? Will we stop to promote business intelligence and predictive analysis as the tools to improve our decision making for large unmanageable organizations? I seriously doubt it. Will we stop the gigantimania and return to 'small is beautiful'? I doubt that as well. Will we stop to hardcode future business decisions in process management? I wonder. People don't even make the connection from this crash to the way we use IT. Thanks again for this blog. You are returning some of my lost faith in the people of this industry!

re: Hal destroys Wall Street

Well, George, as to your third lesson to be learned, here's another example. Unexpected as it may be, it seems highly important to all of us.I am speaking of BPM, enlarging on the topic brought up by Max in his preceding comment ("Will we stop to hardcode future business decisions in process management?").According to a recent paper: "Research into Automation - Identifying the Role of IT in Business Automatization" and a corresponding Whitepaper: "IT-Revisited - The Mistaken Case of Process-Management in Business" (www.eines-alles-nichts.org/mastering-it/Download-WP1.pdf) you, as a CEO, should decidedly say NO to process-management in business.That is to say: the process paradigm, basic to the IT-processor, thus to IT, is most compatible with many procedures within the _departmental stratum_, but fundamentally incompatible with the core procedures in the _business stratum_, i.e. with the intricate features of collaboration and its support by top management. Emphasis, here, is on the intrinsic stratification of the IT-landscape, all too often overlooked or ignored.Example: If investment bankers manufacture (litterally!) their 'structured products' under the regime of some clever BP-model, subsequently transcribed into a BP-application, and finally 'carried out' by a BP-engine, they are going to throw away some important part of the intelligence ('common sense') of their employees, which, otherwise (in case of full fletched collaboration), would have warned them. So ...

re: Hal destroys Wall Street

Bob Lessin, Chairman of the investment bank Jeffries, gives this sequence of events in the saga of how automation killed Wall Street.Step One: Computers enabled electronic trading.Step Two: Electronic trading destroyed the economics of old Wall Street (commissions).Step Three: With the old economics gone, Wall Street was forced to find earnings in new, riskier places.Step Four: These riskier places were computer-derived abstractions like sub-prime mortgage securities.Step Five: I think we all know what happened next...

re: Hal destroys Wall Street

... sure we know, but do we understand?What sort of (border-) line is it that we have stepped over due to the way we handle IT-based automatization? What does human common sense tell us (if we opt to listen) about the nature of the territory we are stumbling in?I feel, common sense, in our days, tells us the Limits to IT-based automation, which is much more than the 'Limits to Growth' of 1972.Common sense, it seems to me, now teaches us the Limits to Automation along and in terms of the borderlines separating production, tele-communication and organization (inluding business).According to my perception of human common sense, the lesson to be learned is this:(1) IT-based automatization is at home in the production stratum, where we find all over (a) physically clear-cut switchboards on which IT-processors can be mounted and (b) procedures that can fairly faithfully be mapped onto process chains and further be translated into programs to govern the IT-processors.(2) In the stratum of tele-communication, however, IT-based automatization is but a guest, since in this stratum automatization is based on quite another kind of tool, such as writing/reading, mailing-networks with all their equipment, converging in the internet. IT-based automatization, while having its poper place, here and there, is secondary.(3) In the stratum of organization, comprising cooperation of people and all of business, IT-based automation is still more of a stranger. Its place must carefully be delimited. The great danger is that 'we' invent and, to some extent, even instal, ad hoc, some fictitious _s e t u p_ on which IT-processors can be mounted and utilized: (a) kind of a switchboard, hiding the intricacies of cooperation and, even more, of collaboration among emplyoees, and (b) a practice, casting the switchboard's handling by managers into neatly arranged, predictable chains of switching steps.If this reads familiar to anyone, she knows (i) a typical case, in which IT-based automation is going to overstep the line, and (ii) the world which it is going to corrupt, and (iii) he understands, why 'without the check of human common sense and understanding, they can do great harm'.Hope my interpretation is not too far from yours, George.

re: Hal destroys Wall Street

Look at what Mark Cuban said. Is the bailout mechanism as opaque as the derivatives that caused the problem? Hmmm . . .http://blogmaverick.com/2008/10/07/when-did-rube-goldberg-take-over-the-country/

re: Hal destroys Wall Street

George I was preparing for a Monday phone call with Debbie Weil & happened upon you YouTube video (don't ya love it when we can hold something hip like YouTube over our kids heads!!)I am founder of a start-up www.buttosnofhope.com working with non-profits and fundraisers -- I am a HUGE fan of Seth Godin (ru?), blogging and web 2.0 as a viable way to reach customers in a new way. Just wanted to say hello --I am going to spend a little time on your blog so you may see some additional comments which I know you will appreciate --keep the faith this stuff does pay just not in the traditional manner!

re: Hal destroys Wall Street

Computing out of control? OK, I'll buy. However, tis greed that killed the beast.

re: Hal destroys Wall Street

When a car's navigation system brings you to the edge of a flooded river, it's time to ignore the machine and turn around. But people have come to trust the technology so much, that they'd rather disengage their brains than switch off the computer:http://www.timesonline.co.uk/tol/news/article707216.ece

re: Hal destroys Wall Street

Not only is this article off base, it's so far detached from the reality of what's going on Wall Street that it's preposterous.I've been building trading systems for over 20 years, and I run a company that provides software infrastructure to hedge funds and the top trading firms in the world. I hate to burst everyone's bubble here, but there's no "Hal" on Wall Street. Traders aren't morons that need to have "securities market 101 presentations." They don't "put trust" in computers to make decisions.The author cites the "massive damage" caused by the reposting of the UAL news story as evidence that computers have gone wild. Yet here's an excerpt from the Washington Post's Frank Ahrens investigation into the incident:JEFFREY BROWN: All right. Then, in the meantime, there is the company at the heart of this, United Airlines. In the end, what's the final tally or the result for the company?FRANK AHRENS: Right. And where can they point? If they assess -- when I talked to them yesterday, they said they hadn't figured out yet if this will have -- in many ways, it was a paper loss. It blips down, blips back up.The real problem is that UAL is a shaky business to begin with, and investors were nervous. The stock went down, and when it was discovered to be a false report, the same day, the stock went back up. But it was HUMANS who did the wrong thing (the article was posted by a reporter incorrectly).In other words, the computers did the RIGHT thing.Moreover, automation is rapidly becoming a bigger part of the solution. All trading firms must submit trade reports to regulators, exchanges have automated surveillance in place, and most trading operations have stringent risk management controls in place in order to comply with regulations. It's not a perfect system, and errors are made, but it's much closer to perfect than the "old days" that the author seems to return to. And because modern trading infrastructure is electronic, we can more easily figure out what happened when things do go wrong, and fix the problem. We couldn't do that 20 years ago, when traders cut deals on the phone without a trace.So not only is automation a force of good, not evil, - it's emerged as one of the TOP priorities for financial firms in volatile market conditions. A recent survey we conducted showed this, and I wrote about here:http://streambase.typepad.com/streambase_stream_process/2009/02/priority-1-an-increased-need-to-automate.html- Mark Palmer, CEO, StreamBase Systems

re: Hal destroys Wall Street

Mark:I'm not advocating that we cut automation. But selling systems-generated products (e.g., subprime mortgages)into opaque markets that evade human comprehension will result in disaster. The way forward must strike a healthy balance between automation and human understanding.George

re: Hal destroys Wall Street

Hi George -Of course automation must be guided and managed - but that's already being done, and done well. By saying balance is "the way forward," you imply that computers are wayward today. And you say we have "crossed a line."My point is that it's not the way forward, it's the now. And computers aren't out of control, in fact, they are helping us gain MORE control than we had in the past.- Mark