Wisconsin Public Television University Place  David Krakauer The Transformative Power of Information  October 1, 2012 UP778   cc >> So, the point about information, and Geoff made this point at the beginning, is it's not conserved. So you can write a book that no one has read, and you've created information, you've created knowledge in some sense. And in the 4th century, a library of many books went up in flames. And so a huge amount of information disappeared, went to zero. That can't happen with energy. So, energy is required to create it, and energy is dissipated in its destruction. So there is some interesting relationship between the two, and part of the challenge that we're facing is to synthesize them. So, what is information? So here's one of the very weird things about information, and we take it for granted, but it's very strange. Here are two structures. They have, physically, almost nothing in common. 

On the right is Big Ben. It has a huge escarpment. The clock mechanism in Big Ben weighs about five tons. It's actually powered by gravitational potential energy. It's got three drive trains connected by three large weights, each of which is several tons. And it has to be wound up again with a motor. On the left there is a little watch, tiny little watch that you can wear on your wrist, unlike Big Ben, that is powered by a battery that lasts several years and is more accurate than Big Ben, and yet both of them tell you the same thing. They both can tell you the time. So how is it that two fundamentally different physical systems can tell you the same information? What is that? So you can sort of ask, what is a timepiece? What is a watch? What is a clock? And you know what it is. The watch is a model of the solar system. 

In fact, the reason why these two objects are telling the same thing is because they're both modeling the same phenomena. They're modeling the rotation of the Earth on its axis and the rotation of the Earth around the sun and sometimes the rotation of the moon around the Earth. So they represent, in some form, an abstraction of a much larger physical system that we express in the form of symbols that is what is meant by telling the time. And there's a watch going off over there now. Okay, now, if you ask what is a solar system, you get a very different kind of answer. And a solar system is in the domain that Geoffrey was describing. A solar system is a bunch of masses that are gravitationally bound to a large or massive star. So, a solar system, in some kind of fancy language, obey energetic conservation laws. So the language we use to describe what a clock is, even though it's a physical system that obeys gravity, it's quite different than the language we use to describe the solar system. And what's happening in that transition from the energetic, mass-based articulation to the informatic. 

So here's what information is. Information is a quantity that is derived through the measurement of a physical system. It's always embodied. And what a measurement does is it reduces our uncertainty in some units of measurement. And the measurement always expends energy. So this is the most general relationship between the physical world and the informational world. So, some of you will be familiar with this, the nerds in the audience. This is, what is it? >> Middle Earth. >> Middle Earth! Thank you! This is a map. This is a map that was drawn by Tolkien. 

He first drew it for his book The Hobbit, and he expanded on it for his trilogy, marvelous trilogy that my wife is obsessed with, The Lord of the Rings. And if you remember the tale of Lord of the Rings, Frodo is told by Gandalf that he has to go from out there in the shire down to Mount Doom to dispose of the one ring. And how do you do that? Well, in some sense you have to start creating a coordinate system. You have to impose a system of measurement on the physical structure. So you might say, there are four quadrants and you need to head down to the lower right quadrant, and the way we normally do that is we say go to the southeast. And that might not be precise enough to get to Mount Doom, and so you zoom in again and you divide that into four quadrants, and you say within that higher resolution quadrant, you go northeast. So in some sense, what you've done is you've taken a physical object that's continuous, you've made it symbolically discrete, and imposed on it a measurement system that acts as a coordinate system that allows you to direct someone to a point in space. Now, the person who actually mathematized the relationship between a physical system, for example a continuous space, and information with Claude Shannon in a very famous paper published in 1948. And this is him. He was extremely playful. 

He was a great juggler. He rode a unicycle. I think he was the captain of the MIT unicycle team, and they have one at MIT. And he also built various odd machines that sort of, in some metaphorical way, captured his interest in information. And here's a machine called the ultimate machine that he built. I'll just show you an animation of it. And you say yes to it, and it comes back to you by saying no. [LAUGHTER] And that's it. And that's actually a replica of the ultimate machine. His was much more beautiful. 

But it captures the notion that information can be expressed in binary, yes and no. So if you go back to the directions in Lord of the Rings, here's Gandalf, and in fact, on Geoffrey's door at the Institute, we have a big image of Gandalf because to some of us he looks a little bit like Gandalf. Geoff thinks I look a bit like Frodo. [LAUGHTER] So here he is. He's giving directions to Frodo. And this is how Shannon described it. There's a signal and a receiver, there's a channel that's sometimes noisy, and you have to get the signal across the channel. And I've already told you how you might do it. You say go southeast, and once you're in the southeast, go northeast. And what Shannon said is you could do it this way. 

You could ask a game of 20 questions. You could say, is it north? Yes or no? If yes, it's north; if no, it's south. And then again, is the next turn east? Yes 1, no 0. And you can encode the same information in a series of yes/no questions which are then written down as a binary sequence. So your computers, which encode information in binary, are doing just that. They are in their memory and in their computations expressing the world in a series of yes/no questions which reduce your uncertainty about the state of the world, which, in the case of a computation, is a solution to an answer, a solution to a problem. And the critical point is that they share a code. That is, both Gandalf and Frodo know that the first bit is north or south, the second is east or west, the third north or south again. And this is what makes information possible. If they didn't share the code, that would be called an encrypted message. 

Okay? So, if you were communicating with an alien and you didn't know the alphabet, the units, I didn't know it was north/south, for example, I didn't know the discrete symbols, you didn't know the syntax or grammar, it would actually be impossible to do so. So even if we were being transmitted a message from outer space, there's a theorem that would tell us in some limit that it would be impossible to decode. And this was actually captured in a science fiction novel by Stanislaw Lem, who wrote Solaris that was made into a movie that we discussed here the other night, a novel called His Master's Voice. This is what he said. He said that it happened that the unknown sender, which is in this case an extraterrestrial civilization, committed a dreadful faux pas because his letter was without introduction, without grammar, without a dictionary. To the best of our knowledge without possessing the code units or the syntax or the vocabulary, the only way to decipher a message was by trial and error. One might have to wait a full eternity. Now, why am I telling you this? I'm telling you this because we face this problem as scientists all the time. Here is an image that you probably all know. It's an image of Watson and Crick that's very famous. 

I think it was taken just after they won their Nobel Prize in 1953. And Crick is on the right, and Watson was the younger of the two. He's looking up. He looks a bit gormless. And he's pointing to the double helix. And they're famous. If I asked you, what did they discover? You'd say the double helix. But the double helix doesn't really matter. What they really discovered was that DNA was a digital system that was symbolically transmitting information. That was the much deeper insight. 

That is, they took a physical phenomena, just physics and chemistry, and they turned it into symbols, into a sentence, in the same way that we took this physical phenomena, and because we know it's a model of the solar system, we turn it into information that we call time. That's why Watson and Crick's discovery is deep, not because they discovered double helix. Now, if you try to interpret the codes, the symbols in DNA as information, we need to take it into Shannon's framework and ask who the receiver is. This is information that's being transmitted, who is it being transmitted to? And we know, right? For example, it's being transmitted to a protein. DNA encodes, through transcription and translation, proteins. And, in fact, science is in the business of intercepting the message and decoding it. That's what we're doing. We're breaking codes. And, in fact, one of our faculty actually, working with Holly in Nuremberg, won the Nobel Prize in 1968 for doing just that. They intercepted the message that was sent from DNA to the protein, and they worked out, essentially, what this clock was telling them. 

What the symbols meant and how they related to protein structure. But if you take certain more sophisticated signaling, for example, written about in the papers every day, diseases for genes, genes for diseases, or genes to personalities and so forth, this kind of thing is harder to decode. There was another Nobel Prize awarded for decoding elements of body form in DNA. These were called the homeotic genes, but it's only a tiny, tiny portion of what a body is. This year, this is an article that was published online in Nature, 30 papers were, in fact, published at the same time, one of which appeared in Nature, I think two in Nature, from projecting code. And let me just tell you what they said. Firstly, the function of around 60% of the transcribed genome remains unknown. That's, by the way, super optimistic. If you talk to my genetics colleagues, they'd tell you it's much larger. So even though since the '50s we've had, in some sense, we've begun to get the full sequence of DNA, we still don't know what it means. 

90% of mutations associated with disease don't even fall in protein code in genes. What we're dealing with here is, in some sense, a message that we can't decrypt. We can't decode it. This is Linear B. Of course it was actually decoded in '51 by Ventris. But this is the challenge we're facing. I think the reason why biology is struggling at the moment is because it's facing the Lem paradox. We actually don't know the language of the genes. All we have is the most elementary units that make them up. So that, for me, is the first challenge of the 21st century with respect to information that is actually recognizing that we need an entirely new project to try and decode, formally, what is being expressed in DNA and in other molecular systems. 

So now let me get to the issue with Geoffrey. How am I doing time-wise? Okay, good. Halfway. Now, this gets a little harder, intellectually, but not that much harder. So, here's that picture again of Shannon's with Gandalf talking to Frodo. And I told you that he sent Frodo a message. He said you need to go down to Mount Doom and this is how you do it. Of course, that takes energy. So here's where the energy starts coming in. You don't communicate a message for free. 

You have to move the muscles in your mouth. You have to activate some neurons in your brain. All of that uses free energy in the form of ATP. Shannon, in fact, wrote down an equation. It was actually called the Shannon-Hartley theorem, and that theorem relates information, which is what this thing is, to power in watts, to the power in a signal. This is actually signal, noise, logarithm to the base two which turns it into bits. The units of this, by the way, are bits per seconds. But Shannon tried to do it. He tried to relate, explicitly, the maximum rate of information flow to the energy availability in the so-called Shannon-Hartley theorem. That was one of the first, I think, more profound attempts to bridge this gap that Geoffrey was describing. 

At the same time, more or less the same time that Shannon was working, we were building our first computers. Big ones. Powerful computers. Here's one. It's called ILLIAC. It weighed about, I don't know. This was first designed by, I think, von Neumann that the series is named after. It weighed about 30 tons, took 200 kilowatts of energy. Extraordinarily inefficient. And so there was a deep concern about the relationship between computation, information, and energy because these things were using so much energy. 

And physicists started to pursue the fundamental limits of the informational energetic universe. And here's one from von Neumann. It's called the von Neumann Landauer Limit. It says what is the absolute minimum amount of energy that you expend to convert one bit of information. That's what that log 2 means. And it turns out to be a very small number because this constant here, Boltzmann constant, is quite tiny. But if you have lots and lots and lots of transistors, you can start to see why the information piles up even though we're no where near this limit. So if you've read the newspapers recently, you've been looking at all these huge centers where they're storing computers, cloud computers, and they're putting them by large bodies of water because they need hydroelectric sources of power and they need coolant. So those limits matter in the design on computational systems. Here's one that I prefer. 

It's a little bit more geeky, but it's even better. And it says, essentially, it's the Bekenstein bound. The Bekenstein bound says what is the maximum amount of information that you could store in a given volume. Where that information would be specified all the way down to the quantum mechanical scale. The maximum. How much information could I store in my finger? And you might say, and it turns out because this is actually a lot of information because this value here, this Planck constant, is actually very, very tiny. And you can ask yourself because of this bound, could we have hard drives that store vast, vast amounts of information, way beyond the amount that we now imagine. There is an object that hits the Bekenstein bound. It actually achieves the maximum possible information density given by the law. And that object is a black hole. 

And so, if you said I want to be able to store every move he ever made. Well, you could do that, but in principle, eventually your hard drive would become a black hole and absorb you. The problem is with these kinds of theorems that were devised to deal with computational systems, is that they were interested in physical limits, and physical limits is not what we're interested in. we're interested in biological limits which are nowhere near the physical limit. So, for example, we know that light travels at about 180,000 miles per second, but a cheetah only runs at about 70 miles per hour. So it's not much use for us to be talking about black holes when we're really trying to understand biology. So we need an insight into these limits. The energetic informatic limits of evolved systems. And for that, actually, you borrow Geoffrey's work. You take the results which these guys have derived for energy in biology, those beautiful laws which all organisms have to obey, and you hybridize them with the insights of Claude Shannon. 

And in doing that, you try to derive things like the maximum possible amount of information an organism could process. The maximum possible amount of information that you brain can store. The maximum rate at which you could transmit information over evolutionary time from one generation to the next. And now I'm going to do that for you, and this is where it gets a little bit nerdy. And so you have to make use of these. Forget this for a second. You have to make use of these power laws, and I'll try to explain to you how this operates. And I'm going to appeal to your familiarity with science fiction for those of you who are not physicists. And so, for those of you who are physicists, this is a light cone. Has everyone seen one of these pictures before? The way it works, essentially, is it posits a present, so all points on this plane are simultaneous. 

They occur simultaneously. And in order to have an influence on the future, all events that you can cause any influence in the future have to diverge away from you at less than light speed. So think about it this way: I'm here at the present, I turn on a flashlight, and the light forms a larger and larger circle as it diverges from me. And that's what this circle is, moving out from the present. Any point outside of that light cone, I can have no cause of impact on because in order to do so, something would have to move faster than the speed of light. And the same is true in the past, everything converging on me has to be within my past light cone. Now, we can take that kind of concept of causality and we can apply it to biology. These arrows just reflect the fact that the diagonal is the maximum velocity of light in a vacuum which, given that time here is measured in seconds and space in light seconds, it would be one light second per second. So here's the biological version. It works like this. 

Here, time is a generation. So it's not chronological time anymore; it's generational time. And here on this hyperplane of the present, space is measured in mutations per generation. That's distance. The distance between us at a simultaneous point in time is going to be the number of mutations per generation. And we can ask, as an organism evolves, if it never changed, it would remain at this point moving forward. But as it evolves, it has to move somewhere within this cone, and it turns out that diagonal is the maximum possible velocity of the evolution. And there is such a thing, and you can derive it mathematically and you find something quite surprising. That microbes are actually moving at the maximum possible velocity permissible by the theory. That is, if you take their results, the thermodynamic constraints, and you combine them with Shannon's theory, it turns out that there are organisms on Earth that are moving at evolutionary light speed. 

We are moving significantly slower. We're living within that cone. So we diverge much more slowly. But here's the very neat thing. This is such a significant constraint that life has had to innovate. So here's my, just a thought experiment for you. How does evolution propagate information? It generates variant offspring, and those offspring that are fitter, survive to propagate that variant, and those that are less fit die. So the way that you would memorize your mother's name by an evolutionary process is you'd have a million people, one of whom knew it, all the others did not. They would die, the one that knew it would propagate it, and so forth. This is very inefficient way of learning because the way you learn in evolution is through death. 

And life can't deal with that, or it can but it's found a solution. It's found a solution of moving faster, informationally. It's actually gone through a series of informational revolutions that allow information to be propagated more quickly than the fundamental theorem would allow. And this is how we do it. We do it with epigenetics, we do it with cell surfaces that are responsive to changes in the environment, and we do it through brains. So, when Geoffrey introduced the notion of complexity, why is there complex life on Earth, the reason why is because we've overcome, through the evolution of these mechanisms, fundamental constraints that are built into the evolutionary process. Now, those systems we have barely, barely a clue how to decode. And so the challenge for us now is to try and work out the sort of Watson-Crick like alphabet but then the -- genetic code for a brain. Now in 1956, here's a book. It's called The Handbook of Biological Data, and it had 584 pages, 445 tables, and it included genetics, physiology development, and ecology. 

Well, we're going through a huge cultural exponentiation in the amount of information we have, and we couldn't fit even genetics within the Library of Alexandria. So we have a double challenge, right? One, we don't really have a principled way of decoding messages where the units are not known. And secondly, we have a total overwhelming amount of data to deal with. And the question is, how are we going to solve it? Should I end on your note, which is on a pessimist, or should we end on the optimistic note? And I'm actually somewhere in the middle. I'm sort of agnostic with respect to pessimism and optimism. And what people are telling us is this is going to come to our rescue. That the fundamental means of overcoming both the vast quantity of data and the fact that we don't know how to find regularity in data is going to be computing. That computing is the answer. And I actually don't know, but perhaps in the conversation we can discuss why I'm uncertain. I will just point out one or two things in my final slides. 

Firstly, there's a very, very intimate relationship between information and computation. If you remember information, it was when Gandalf sent a signal to Frodo which he didn't want to be in any way corrupted. Computation is a two-way informational process where Frodo will say am I in the right place? And Gandalf will reply yes or no. That's a computation. It has a slightly different quality. One is about deciding the output of a question, and the other one is about transmitting a question reliably. And you can actually integrate the two in another theory which tries to combine the theory of computation with the theory of information as opposed to theories of energy. I put that picture up in the background because he's another hero of ours here, Alan Turing, and many of you will know him from his fantastic mathematical work and mathematical logic on the problem of undecidability. That's actually a question where you pose that you can't answer it. But he also applied his logic, contrary to the expectations of people who think that basic science is worthless, to help win the second World War. 

And he worked in Bletchley Park. He was hired, actually, in 1939, very soon after the war started as a crypt analyst. And he was drawn to this machine, the enigma machine, that was essentially being used at the time by the Germans to position submarines to attack boats. U-boats were positioned to attack boats. It's a polyalphabetic rotor-based cipher system. It's a fascinating code. And because no one else was working on it, Turing said, well, I'll do it. And he cracked it, and in the process, he was able to identify the positions, reduce the uncertainty of the spatial position of U-boats. And he was actually knighted for this work. My claim is that in the 21st century, we all have to become hackers. 

That, in fact, this is the 21st century challenge. How do we do what Turing did with the enigma machine with evolved enigma machines that we call nature? And so this is the challenge. We need a new theory, as Geoffrey was saying, that integrates effective limits into our informational theory. And I would claim that it would actually fundamentally revolutionize, more or less, everything because if you think how far we've got with just the alphabet of DNA, where most of the phenomena remains mysterious, if we could really pull it off, I would claim that there's potentially significant advantages, both in practice and intellectually. Thank you. [APPLAUSE]