How to do Science

Ok, that's rather an ambitious title. I will, in fact, only cover a bit of how science is done. But one of my main themes here is how science is done, what the thinking is like, what the language is like and why. I would like to convey a better understanding of what scientists do and how to understand what they have to say.

For today's discussion, I want to start with a really excellent essay Richard Feynman wrote, based on a commencement address he gave in 1974. Feynman was one of the nation's leading physicists from the end of WWII until his death in 1988. He was a Nobel prize winner and was perhaps most famous for uncovering the problems with the o-rings that were responsible for the Challenger disaster.

Read the whole essay, but there is one part in particular that I want to discuss here. At this point Feynman is talking about one central requirement for doing good science.

That is the idea that we all hope you have learned in studying science in school -- we never say explicitly what this is, but just hope that you catch on by all the examples of scientific investigation. It is interesting, therefore, to bring it out now and speak of it explicitly. It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty -- a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid -- not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked -- to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can -- if you know anything at all wrong, or possibly wrong -- to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.

In summary, the idea is to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.

It is certainly true that scientists do not always live up to this ideal of presenting all critical evidence. Scientists are human too and sometimes fail to live up to the highest ideals. However, doing good science and gaining the greatest reputation in science requires following this ideal as closely as at all possible. Generally speaking science follows the above ideals pretty well.

This requirement of presenting contradictory evidence, however, is not typical in most other activities. Neither a lawyer not an advertiser nor a salesman would do too well if he or she spent too much time pointing out the possible flaws in his clients case. Because of this characteristic the way that scientists talk about their work often seems a bit strange to non-scientists.

In almost all areas of science there exists some contradictory evidence, or at least evidence which doesn't fit the current theories as well as we would like. So there is almost always some contradictory evidence to recognize and therefore some uncertainty in our conclusions. But also realize that in some cases there is very little that doesn't match up just right, and even then the mismatch is small, and in other cases there are large puzzling problems and other cases that fall in every degree of uncertainty between these extremes. Therefore one of the striking characteristics of scientific speech is an effort to express the degree of uncertainty. Hence the frequent use of qualifiers and so called "weasel" words.

See, in most non-scientific venues people want to come to some conclusion, even if they are not absolutely certain it is correct, and yet act upon it with some action in spite of any uncertainty. In those circumstances "weasel" wording is undesirable. In the scientific arena, they are an essential part of communicating the degree of uncertainty and are necessary for reaching the best, most likely to be correct, conclusions that we can reach.

The Challenger disaster and modern management

Richard Fynman served on the review board which studied the causes of the Challenger disaster in 1986. One of his chief observations was that there existed a huge disconnect between Engineers who worked closely with the spacecraft and Management. While the Engineers were aware and knowledgeable of the many risks with the spacecraft, Management was generally much more ignorant. Management had a very rosy view of the status of the craft and it's ability to be launched safely. The Engineers were a great deal more skeptical. And detailed investigation demonstrated that, not surprisingly really, the Engineers view was much closer to reality. So something about the management of the program prevented accurate communication of the status of the spacecraft from the ground up to Management.

The same problems still existed for the Columbia disaster in 2003. It seems to me that much the same problem could be found at the FBI and CIA prior to 9/11. There was a great deal of information available to front line members of those organizations which simply did not get up the chain of command.

The point here is that there seems to be a fairly widespread problem with a failure to communicate from the ground up a chain of command. Furthermore, the problem seems to be that management is resistant to hearing about problems. Management is isolating itself. Unfortunately, this isolation frequently results in disaster.

This leads to one of the reasons for the strong dislike of President Bush. Far from trying to solve this problem, he intends to exacerbate it. This is especially evident with the recent cabinet shakeup. The attitude of this administration is not that we need to improve communication from the front line to the top, that the folks at the top need to do a better job of listening to those on the front line. Rather this administration apparently feels that the only problem is that front line folks talk at all. This administration acts as if it will decide what is going on in the world and the role of front line folks is only to support them and take the fall if disaster occurs. This is an awful system of management.