9UNP0167SD_Nora_Cate_Schaeffer_WordPress

- Welcome everybody, to Wednesday Nite at the Lab here at the Biotechnology Center at UW-Madison. I'm Tom Zinnen. I work at UW-Madison here at the Biotechnology Center of UW-Madison, and also for UW Extension, Cooperative Extension. We do Wednesday Nite at the Lab every Wednesday night, 50 times a year. It's also broadcast on Wisconsin Public Television. Tonight, I'm delighted to introduce to you Nora Cate Schaeffer. She's the director of the UW Survey Center. She'll be talking about "20 Years of Taking the Pulse of Wisconsin." This is the 20th anniversary of the founding of the Survey Center. 

It's a great pleasure to have Nora Cate here, in part because my wife works at the Survey Center. That's how I found out that this was the 20th anniversary. I'm very pleased that Nora Cate would give a talk, telling about her research and about the work of the Survey Center, and what they hope to do over the next few years looking into the future. Please join me in welcoming Nora Cate to Wednesday Nite at the Lab. 
(applause)
 
- Thank you. When Tom invited me, he assured me this was a very interesting audience that I would be meeting here. And I met some of you on the way in. And certainly, you live up to your advance press. 

I'm really very delighted to be here. This isn't a kind of talk that I give very often, so you'll just have to, I hope be generous in your assessment in what I have to tell you. As Tom mentioned, this is the 20th anniversary for the UW Survey Center. And I'm just going to say who we are before I launch into what I'm going to be talking about. The Survey Center is a branch, or a unit within the College of Letters and Sciences. We serve primarily faculty at the University of Wisconsin. We also serve investigators in the state of Wisconsin. And we'll be talking about at least one of those studies today, as well as one of the studies we do for a faculty member in Sociology. We also conduct surveys for other people, investigators all over the country. We do not, however, do political polling. 

So even though polls are in the news a lot right now, and many of you may have wonderful questions about the polls, I'm afraid I won't be able to answer questions about the polls. I'm afraid I won't be able to answer those questions. However, one of the topics I picked to talk about tonight might inform your own thinking about how you view the results of some of the polls that you see in the news. So we also don't do any marketing research. In fact, we don't do any proprietary data collection at all. All the data that we collect ends up in the public domain one way or the other. We do a kind of survey that I would describe as social research surveys. So they're different than polls. They're different than other kinds of data collection that looks sort of like the kind of survey that I'm talking about. I think you'll have, by the time I'm done, I think you'll have a sense of the flavor of the kind of work we do at the survey center. 

What I have up on the screen is an image of a postcard that we mailed out to people in the university community announcing our 20th anniversary celebration. As we were trying to decide what to do to observe our 20th anniversary, I said, well, we're survey geeks, let's have some short courses. So what we did was have some short courses. They were, I'm glad to say, a huge success. We had more people than we could really accommodate in the seats we had available. The survey center staff were just amazing in the way they managed to produce and put on these short courses. But you can get a sense of what the people who work at the survey center look like. And the wallpaper behind the images, which you'll see again at the end a little more vividly, lists some of the principal studies that they've been involved in. Now, when Tom asked me to talk, he suggested I talk about 20 years of taking the pulse of Wisconsin. And as I mentioned, this isn't the kind of talk I usually give. 

I usually talk about my own research, which isn't particularly interesting to most people. It involves looking at what interviewers and respondents do when they interact with each other. So it involves looking at things like who says "okay" when and when does somebody inhale, and things like that. So we'll do a little bit of that at the end. I just snuck in a little bit of it. But in order to talk about something that might have a little bit more general interest, I thought I would say, and trying to keep sort of in the spirit of Tom's request, I thought I would talk a little bit about some of the main surveys that we do and the issues that they raise for people, for you as people reading survey data, reading about survey data or results of the surveys. Some of the questions you might have about how survey researchers approach certain kinds of problems that surveys face. Because this is Wednesday Nite at the Lab, I thought I would talk a little bit about surveys as a scientific enterprise. Then we'll look a little bit at some of the Wisconsin surveys. So first, science in the survey lab is the first thing I want to talk about. 

Places like the University of Wisconsin Survey Center are often called survey laboratories. That's just an informal term that we have for them in the profession. That's one of our pieces of jargon. And it gives us the feeling that we're doing real science, too. And that's a feeling that we like to have. But there are lots of different kinds of scientific questions a person might have. For example, in other kinds of presentations that you've had here, are probably more likely to talk about these kinds of questions. So for example, a researcher might conduct a study to try to demonstrate some kind of cause and effect. That would be a very common kind of scientific question for a researcher to examine. Less common, but also scientific, is something like trying to figure out how a speaker generates the next sentence that they're going to say, given what's already been said in a piece of talk. 

And that's not experimental, but it's definitely scientific. And there are lots of researchers doing artificial intelligence, and trying to design better automated voices to talk to you on the telephone, who are interested in exactly that kind of research. Another kind of scientific goal that someone might have is something like assessing the clinical status of a patient who's in front of them. How sick is this patient, and what are they sick with. So those are all different kinds of scientific questions that have nothing to do with what we do at the survey laboratory. What we do is try to describe calculation. And it's a very specialized, like these others are, kind of scientific question to take on. Now, in order to describe a population, you really have to meet two criteria, or satisfy two sort of things. There are two things you have to do. The first thing you have to do is to be able to represent that population in some way. 

And the technology that we have for representing a population is called a probability sample. And I'll say a little more about that on the next slide. The other thing that we need in order to describe a population is a way of measuring the units in that population. And what we mean by measuring is a way of getting comparable information from each unit in our sample. So if I ask person number one in my sample a question, I have to ask the same question to person number two, so that I can compare their answers with each other. If I don't ask them the same question, their answers won't be comparable. And so we have technologies for trying to represent the population. And we have technologies for trying to take measurements of the population. Let's go back to sample and talk about that for a few moments, because this is really where we're going to spend most of our time. The idea behind drawing a probability sample is that we want to describe a population. 

But we can't see the whole population, we can only see a part of the population. That's our sample. Furthermore, we can never know for sure, because our sample is only a part of the population, whether or not that sample is like the population. However, even though we can't know if the sample is like a population, we can know the process by which the sample is drawn. And probability sampling tells us a description of how the population is drawn into our sample, how units from the population are drawn into our sample. In a probability sample, each unit in the population has a known non-zero probability of being selected into the sample. As a result of that, I can say the sample is either like the population or it's different from the population by chance. And that is the key to the statistical inference that researchers do on a sample, and which I teach about when I teach Introductory Statistics, and which we won't be talking about at all tonight. That may be fortunate or not. But the fact that we know that the population, that the sample is like the population except for chance, really is a very powerful tool. 

It's a good part of the science that goes into scientific sampling and making surveys scientific. The probability sampling was really a wonderful idea. In fact, it's so wonderful that it's like a percentage. When you look at it closely, it's so astonishing. You can't imagine, I can't imagine how anybody even thought of it. Probability sampling is one of those inventions that, to me, is really just incredible. But even though it's a great idea, in order to use it, you have to do it in the real world. And the real world, of course, is a much messier place than wherever survey statisticians devised their formula. So when I say every unit in the population has a known non-zero probability of selection, some of you might have been wondering, well, how the heck do you do that. How can you know that each unit, we don't have a list of the population. 

Well, we kind of do. There are a few countries in the world where there are population registers. And there really are lists of the population. The United States is not one of those countries. And it has, there are lots of good historical reasons for it, but it makes our lives as survey researchers more complicated. One of the things that we have to have in order to draw a probability sample is a good list of all the units in the population. If the list of units in the population isn't complete, we say we have a kind of threat to the validity of the results and the conclusions that we might draw from our survey. One of these threats to validity we call coverage error. Coverage error is a kind of threat to validity that results when this list from which we draw the sample systematically excludes part of the population. And some of you might be thinking, oh, like telephone samples that don't include cell phones. 

And I would say yes, that's exactly what we're going to talk about in a moment. Another kind of threat to validity that can arise with probability sample is that when we select everybody into our sample, we need everybody to participate. When people don't participate, we call that non-response error. So we have these two kinds of errors that threaten validity. There are also issues of measurement error, as you might suspect from my earlier comments. And we'll come back to that in just a little bit. Now in order to talk about the first threat to validity that I mentioned, that is coverage error, we're going to talk a little bit about one of the principal surveys that we've been conducting at the Survey Center over the years. The Behavioral Risk Factors Surveillance System surveys, also knows as BRFSS, which doesn't sound very good. But that's what we call it. The Behavioral Risk Factors Surveillance Systems surveys are really, they're not one huge survey. 

It's really a compilation of surveys conducted by each of the 50 states, and some of the other units, like District of Columbia, Puerto Rico, and so forth. So the Centers for Disease Control in Atlanta has developed a protocol, a set of questions. Then each of the states then goes out and conducts its own survey. The results are compiled, both at the state level, and then also made available as part of an entire surveillance system through the CDC in Washington, DC. So the state health departments in the 50 states and other units conducts, each of them conducts a separate survey. And the goals of these surveys are to monitor behavioral risk factors. That is, things that people do that put them at risk for compromising their health, such as smoking, binge drinking, and so forth. The surveys are also used for a kind of surveillance. For example, to monitor who's getting flu shots, the spread of flu. It's one of the ways that the spread of certain infectious diseases is sometimes tracked. 

But mostly the flu shots themselves. Now the Behavioral Risk Factor surveys, which are conducted in the 50 states, each have a similar design. The one we conduct for the state of Wisconsin has as its target population, the population we're trying to describe, adults in the state of Wisconsin. In order to do this, the list that we use, which we call a sample frame, is a list of phone numbers for people in Wisconsin. We call this a telephone sample frame. And this list of phone numbers is generated by a method known as random digit dialing, or RDD, is the jargon we use. We talk about having an RDD telephone sample frame and RDD sample design. Now, when these survey designs originated, they really were random digit dialing. That is, people, investigators, would get the initial prefix and area code, and then randomly generate the last four digits. When you do that, you get a lot of numbers that aren't working, or that belong to businesses and so forth. 

And it's very expensive to call through all those numbers and try to figure out whether it's a household that you really want to reach. So modern designs are a little more complicated than the originals, but they maintain that concept of generating the last four digits to some kind of prefix that has a high probability of being connected, of being answered in a residence. The BRFSS surveys are pretty big. We sample about 26,000 lines during the year. And we interview about 4,000 households. But the sample is spread over all 12 months of the year, so there's a little miniature sample each month. That way, you can sort of track certain things that might be changing in the population over time. The interview is about different kinds of health behaviors and certain kinds of health conditions, such as asthma. And the clients for the survey include the Wisconsin Department of Health and Family Services, the DHFS, as well as the Centers for Disease Control. The interview is about 22 minutes long. 

And the interview is conducted by telephone. I'm not going to go through all the details of the sample design, but when we call a household, someone answers the phone. We find out how many adults are in the household and we randomly select one of them. And that's the respondent, that we talk to, and conduct this 22 minute interview with. Now, I'm going to focus just for an illustration of coverage error, the first of the threats to validity, on the behavior, the risk factor of smoking. The interview includes some questions to determine whether or not people smoked in the past, and whether or not they currently smoke. So the second question on the screen is about current smoking. Do you now smoke cigarettes every day, some days, or not at all? And if the respondent says that they smoke cigarettes every day or some days, they're considered to be a current smoker. I already mentioned that if we want to, in fact, do something like describe what proportion of adults in Wisconsin are current smokers, using a telephone sample in which we're randomly generating phone numbers, we need every phone number that is connected to an adult in Wisconsin, to be somehow part of our sample. 

BRFSS, however, several years ago made a decision that cell phones would not be included as part of the BRFSS sample. So if we reach a phone number, one of the first questions we ask is if it's a cell phone. And if the person is talking on a cell phone, we don't interview them. There are lots of reasons for this. One reason is that lots of times people are talking on the phone while they drive. And we don't want to try to conduct an interview with somebody while they drive, especially if you're doing a survey about health risk factors, it seems sort of against the spirit of the enterprise. So, BRFSS has, in the past, prohibited conducting interviews on cell phones. As things have developed, however, this kind of can make us a little bit nervous for at least two reasons. We can start thinking about well, how does excluding households with cell phones potentially affect the validity of the conclusions we might reach from a survey like a BRFSS. That leads us to two questions. 

The first is, what proportion of people in Wisconsin have cell phones. Because if hardly anybody has a cell phone, or if the people who have cell phones also have land lines, it doesn't really matter, because we'll reach them on the land line. So it's really the cell phone only households that are at issue. And the question is, what proportion of the population lives in households that only have cell phones. Those are the households that we're excluding from that sample. The other question is, are the household that are cell phone only households different from everybody else. If they're the same as everybody else, the fact that we've excluded them might seem impolite or undemocratic, but it won't compromise the validity of the results. So those are the two questions that we then take to the next set of results. And this is too small to see any details, but I'll tell you what the lines refer to. This is a graph produced from the National Health Interview Survey, which has taken upon itself the task of trying to monitor what proportion of households are cell phone only households. 

In this case, in this particular table, what proportion of adults live in households that have only cell phones, and what proportion of children live in households that have only cell phones. They are also monitoring the proportion of adults and children who live in households that have no telephone at all. So what we see here beginning in 2004, or around this part of the graph, and ending in 2007 up here, the bottom line we'll do first. It's fairly straight forward. This is the proportion of children in red and the proportion of adults in blue, who live in households with no telephone at all. You can see that that proportion has been between 1-1/2 and just under two percent, pretty steadily. That's been true for a long time. The line that has a very steep slope is the line that's of more concern. That's the line that describes the proportion of adults and the proportion of children who live in households that have only cell phones. And you can see that since 2004 that proportion has increased from about four percent, a little under five percent, up to almost 14 percent in 2007. 

So the proportion has been increasing pretty substantially and pretty quickly. Are they different from everybody else, these people who live in households that have only cell phones. Well, actually they are. It turns out that if you look at the age distribution and ask yourself, of the people in each of these age groups, what proportion live in cell phone only households, for people who are under 30, almost one-third of them live in cell phone only households. So the people who only have cell phones and no land lines are more likely to be young people. Or young people are disproportionately likely to live in cell phone only households. Similarly, people who are poor or near poor, are disproportionately likely to live in cell phone only households. Now, there are a number of other characteristics that I could have put up on this table. But most of them are probably associated with either being young, or having constrained economic circumstances. So for example, people who rent are more likely to live in cell phone only households. 

They're also more likely to be young and to have less money than people who live in houses, who are buying their houses. So these two characteristics kind of summarize some of the key features of people who live in cell phone only households. Now you might be wondering how we know this about cell phone only households. Well, the National Heath Interview Survey uses a different sample frame. It uses what's called a national area probability sample, and conducts face-to-face interviews. So the list that the National Health Interview Survey uses to reach everybody in the country is basically a map of the country, which includes everybody. So people are sampled by geographic area, and once you get to a low level of geography by address. And that's how we manage to reach everybody in the country, even though we have no list of everybody in the country. But this area probability sample that uses face-to-face households is a way that you can get information about which households only have cell phones, information that you can't obviously get by using the kind of RDD, or Random Digit Dialing System of land lines that the BRFSS uses. 
(inaudible)

 They have a series of questions in that National Health Interview Survey. It's actually a whole module, a set of about 15 different questions that they ask. And there are actually several different versions of the questions that are used to ascertain whether or not people live in cell phone only households. And sort of to describe this cell phone structure of households generally. Do people have personal cell phones, and so forth. So they get very complicated very quickly. And when I teach Questionnaire Design during the summer at the University of Michigan, one of the exercises that my students do is to work on the cell phone questions from the National Health Interview Survey and from another version used by the Pew Center for research on people and the press. They're really interesting. It's a really challenging set of things to try write survey questions about. Did you have a question? 

- Are non-responses considered acceptable? 
- Let's wait till we get to the Wisconsin Longitudinal study to address that. 
(inaudible)
 It depends on the survey. It depends on the design. And it depends on what variable you're looking at. There's really no hard and fast rule. That's the short answer. In general, you like to get as many responses as possible, because that way, we don't have to worry about how the non-respondents are different from the respondents. But there's no hard and fast rule. 

There's another question. 
- How do you account for what might be a bias from the time of day that you choose to make the phone call? 
- In terms of respondents, or in terms of quality of the answers? 
- In terms of respondents. 
- Yes, I will. Thank you. Say it again. 
- In terms of respondents that you're interviewing. 
- So how do we take into account the impact of the time of day that we make the phone call on whether or not people agree to participate. And the way that we take that into account is we try all times of day. 

So when we do a telephone survey like BRFSS, we may call every phone number 15 or 20 times, at various times of the day. And one of the kinds of procedures that distinguishes research of the kind that we do from research of other kinds, is that we try as hard as we can to reach the people that we sample. So we keep calling at various times of the day. And if the respondent doesn't seem to like to talk to interviewer number one, we have interviewer number two, who might have a better fit with the person call, and try and persuade them to participate. Other questions? Yes? 
- Have you looked at the use of caller ID now, because I think that would affect your statistics similarly to cell phones, because people, if they don't know your number, may not answer. 
- There is some work on caller ID. Yes, the question is, is there any information about the impact of caller ID. And the answer is there's some information, but it's not as systematic as the information we have about the possible impact of cell phones. 

So we do believe that caller ID and all those ways that people have of insulating themselves from unwanted calls makes it harder for us to contact people. Other questions? Yes? 
- You would think that in our hurly, burly world, to get anybody to be willing to talk on the telephone to an unknown person for 22 minutes, would be extremely difficult. How do you get anybody to say yes, I'll be glad to talk to you for 22 minutes? 
- The question is, given everything that people have to do, and the pressures on people, and you could add to that list, right? The suspicion that people have that this might be part of a scam or part of a fraud, you can add all kinds of things to all the kinds of reasons that people might not want to do it. How do we get people to participate. It's sheer altruism on the part of the respondents, is basically what it comes down to. And it's one reason that people like us who do social research surveys have better success than people doing polls, or people doing commercial research. 

And that is because there is, there are a lot of people who fundamentally are willing to contribute to a research enterprise that aren't willing to participate in a commercial enterprise. And as a result, we have, for the BRFSS, for example, our response rate run, they're much lower than they used to be. But they're still between 50% and 60%. 
- Is it a bias? 
- Yes, it is. And we'll talk about that when we get to the Wisconsin Longitudinal Study. So the question is, does the non-response introduce a bias. And the answer is yes. And we'll talk about that a little more in an interesting way. I have an example that I hope you'll find interesting, although it's not as detailed as the one that I have about cell phones. 

Other comments? Okay. 
- My husband never answers any surveys. And my daughter said that in high school, when they had these written social surveys, many of them answer just false maybe, or just fill in the dots. How do you account for that? 
- So the question is, what about people answering falsely. In addition to the problem of non-response, what about people giving false answers. For this kind of survey, we don't worry about false answers too much overall. And I'll qualify that in a moment. For the most part, if people are going to donate half an hour of their time to talk to you on the phone, they're not going to bother themselves with not telling you the truth as best they can. 

The partial exception to that-- That isn't to say that answers are accurate, however, because people's memories aren't perfect. Even though people are willing to give you half an hour of their time, they might not be willing to say, hang on a minute, let me go upstairs and pull out all my financial records and look at exactly how many times I went to the doctor in the last year. That could take several hours to figure out. So people estimate their answers. And those estimates won't be completely accurate. So even when people are motivated and willing, there are these sources of inaccuracy. There are also sources of inaccuracy that arise when you ask people questions that they really don't think are any of your business, such as, how many times have you been arrested for drunk driving in the last year. Now, that's probably something most people who were arrested for drunk driving, don't really talk about all that often. And if anybody on the street were to ask them, they'd say zero. Because that's a polite way of saying it's none of your business. 

That's private information. So when we ask people questions for which they think the answer is private, then sometimes people will give us their public answer rather than the private answer. And that can lead to problems, because sometimes the surveys are used to estimate things where it really is important to have the private answer that people don't often tell other people. And that's an ongoing problem. And one of the things that survey researchers experiment with, is ways to help people feel safe to give their private answers in a survey interview. So one of the things we do is usually try to collect information like that using self-administered forms, even when an interviewer is present, so that people don't have to say out loud, for example, how many times they were arrested for drunk driving. Yes? 
- When you get surveys in the mail, and always from private companies, where they enclose an incentive, like a dollar bill or something. I suspect that probably enhances people's interest in cooperating. 
- So the comment is that sometimes you receive surveys in the mail that have an incentive attached to them. 

And that probably increases cooperation. And it does. And in fact, it's extremely cost effective. Every once and a while, someone will discover that we're giving people incentives. And not only are we giving people incentive, but we're giving people incentives before they participate. So we send out a questionnaire, and attached to the questionnaire is a five dollar bill. And there has been at least, in one instance I can remember here in Wisconsin, an outcry that we were wasting taxpayer money by doing this. Whereas the research indicates very clearly that that the's cheapest way to get people to participate. Because if you send that pre-incentive, you're more likely to get the questionnaire back the first time. You're less likely to have to mail the replacement questionnaire. 

And by the time you've mailed the replacement questionnaire, you've spent more than five dollars right there. And if you're going to send them five dollars after they participate, you have to keep track of who participated and make sure it gets to the person. There are a lot of complications that ensue. So it turns out in many circumstances, not all circumstances, it's more cost efficient to send the incentive in advance. Okay, so I'll back to the cell phone. So the National Health Interview Survey. The second question, one question we had was how many people are there with cell phones. So there's an increasing number of them, so it's worrisome. From the point of view of the validity of the results of the survey like BRFSS. The second question is, are those who live in cell phone only households different from the people who live in households with land lines. 

And the answer is, in some ways, yes. And here, we can focus on the second line of this table. The proportion who are current smokers. So what this table tells us is, that in land line only households, 18% of respondents are current smokers. But in cell phone only households, 28% of respondents are current smokers. In no phone households, the percentage is even higher, 36%. But remember, about under two percent of households are no phone. So even though they're very different from the people with land lines, there aren't very many of them. I'm sorry? 
- How do you discover the two percent of no phones if you can't call them up? 

- The way we discover the percent of no phones, when we can't call them up, is to do a face-to-face interview. So using, not a telephone number sample frame, but a sample frame that's based on geography, so that everyone is included. The information from the National Health Interview Survey is based on face-to-face interviews, using a different kind of sample. Okay, so we have these smokers in cell phone only households of whom there are more than there are in land line households. If you go to the Centers for Disease Control Web site, you can get access to the BRFSS data for all of the states. And you can pick any of the behavioral risk factors and have BRFSS produce for you a nifty little map that looks kind of like this. I zoomed in on Wisconsin, but the information is produced for the country as a whole. And you can see from this map, here's where we are in Wisconsin. And Wisconsin has between 19% and 21% of adults in Wisconsin households, 19% to 21% say they are current smokers. So a little more than in Minnesota, neighboring Minnesota, where it's between 17% and 19%. 

And a little less than neighboring Michigan, where it's 21% to 24% for current smokers. But when we hear the information that in Wisconsin, 19% to 21%, where'd my cursor go? Oh, it's gone. Where 19% to 21% are current smokers, we might wonder, well, what's the impact of omitting cell phones from that estimate. Because cell phone only households are not reflected in this document. And we know already that cell phone only households have a higher proportion of smokers in them than do households that have land lines. So if we go to the BRFSS Web site for Wisconsin, we can make a nice little graph like this. It shows the percentage of adults who are current smokers, beginning in, this goes from 2000 to 2006. I couldn't get the labels on one axis to come out exactly the way I wanted it. We see from this graph that over the time period that we saw using the National Health Interview Survey that the proportion of households that are cell phone only is increasing. 

The proportion of smokers estimated in Wisconsin, using land lines only was decreasing. Now, we could ask ourselves, is smoking decreasing for real, is it really decreasing, or is this an artifact of our having omitted households that have cell phones only from our sample. So is this a real decline, or is it a result of increasing coverage error. And it's a question we don't have an answer to. I can, however, tell you that I looked, in preparing this talk, I looked at several behavioral risk factors. And the information on current smokers made the nicest non-coverage story, because it showed the most change over the period. If I had put up binge drinking, I wouldn't have had quite as nice a story to tell. So even though this picture makes it look as though, wow, there's almost certainly coverage error that's been affecting these estimates, if we had looked at another one of the risk behaviors, it might not look quite so clear cut. Now, the BRFSS people at the Centers for Disease Control in Washington DC have thought about these issues pretty hard. So now, in the current year for the Behavioral Risk Factors Surveys, many of the states are undertaking experiments with trying to include cell phone only households in order to try to reduce any possible coverage bias. 

So this is kind of an example of the way survey researchers think about the kinds of problems that are introduced by a possible threat to validity from the increased use of cell phones, and from only cell phones by certain households. Does anybody have any other questions before we leave coverage error, and go on to non-response error? Yes, Tom? 
- Is there a concern about things like Vonage and Skype? 
- So, is there a concern with things, basically people using voice over IP as sources of their telephone service, and the bias that that might introduce. There is some concern about it. But I think the thought so far is that the number of households affected is probably still too small to have any big impact on the estimates we make from our surveys. But it's the kind of thing that you would want to monitor, because at some point it could become much bigger. It's also very hard to write general questions about it. And one of the problems that's suggested or challenged by Tom's question is that as the options for getting telephone service proliferates, the number and complexity of the kinds of questions that you have to write in order to track it just, you know, that whole measurement problem, trying to measure the problem that's causing this coverage error, or potential coverage error becomes, itself, a kind of challenge in measurement. 

Other questions? Yeah? 
- A classic example of this was the 1936 presidential election where the Literary Digest did their poll. They thought that the republican candidate would win overwhelmingly. The problem was that they were using telephones, which only the wealthy people still had in those days. They were speaking mostly with republicans, so they predicted a republican was going to win that election. 
- There were definitely sample frame issues in that particular situation. Other comments or questions? Well, let's look at, turn to another survey. The Behavioral Risk Factor Surveillance Survey is one of the kinds of surveys we do. 

One of the biggest surveys we do at the Wisconsin Survey Center is the Wisconsin Longitudinal Study. This is a study whose principal investigator, Bob Hauser, is a faculty member in the Sociology Department, where I'm also a faculty member. The faculty director of the Survey Center is just a part-time job for me. The Wisconsin Longitudinal Study is a very large study of an extremely specialized and complicated kind. So in the BRFSS that we looked at, the sample frame, I told you it was this RDD, this Random Digit Dialing sample frame, this list, from which we selected our sample. For the Wisconsin Longitudinal Study, the sample frame is completely different. It's the list of people who are enrolled as seniors in high school in Wisconsin in 1957. So it's what we call a list sample. And they were selected in 1957. We've been interviewing them periodically ever since. 

It's a kind of design we call a panel study. We select a group and then we keep going back to them and saying, can we please talk to you one more time. It will only take about five hours. 
(laughter)
 And fortunately, this study was done in Wisconsin and not some other states that I won't name, because some of you may be from them or have loved ones who live there. But in Wisconsin, you actually have a chance of making something like this work. And you'll see, well you won't see, because the picture is too small. But I'll describe to you the success in just a moment. So the frame is what we call a list sample. The interview covers a wide range of topics. 

In our last round, our telephone interview was more than 70 minutes long. And many people completed it in more than one session. But they completed really, a very substantial and very demanding survey interview, contributing their time and information to the scientific enterprise. In addition, once they hung up the phone, we then sent them a self-administered questionnaire to fill out. And that, you know, one of the kinds that you usually get in the mail, it was all about 50 pages long. Who is in here? Well, I've got a couple of pictures from the WLS folks, the Wisconsin Longitudinal Study. And they referred to themselves as a survey of the Happy Days cohort. So they include photos, actually, it looks like Professor Hauser got fan mail from the cast of Happy Days here. And out of the original bunch of people enrolled in high school in Wisconsin in 1957, about 30,000, they took a one-third sample. 

So they sampled about 10,000, what we call graduates from the high school class of 1957 in Wisconsin. This slide simply lists, and kind of summarizes all the data collection that's been done on this sample since then. We won't go through it in detail. I just wanted you to get a sense of the enormity of the enterprise. So it started in 1957 with a survey of the graduates. And then not content with interviewing the graduates, we decided, or the investigators actually decided over the years to investigate almost everybody who knows them. So the parents have been asked, sent a postcard survey in 1964. Siblings have been interviewed. Spouses have been interviewed. There's some people would like to try to interview their children. 

And so, the plan to use the information that these people have been donating to science to really enrich the data further continues. And right now, there's been funding from the National Institute on Aging to conduct interviews with the graduates again beginning around 2010. 
(inaudible)
 Oragene DNA kits refers to data collection that's being done to collect saliva from members of the sample in order to get information about DNA that researchers will then use to look at the relationship between DNA and certain kinds of health outcomes. So it's really a massive undertaking. And the respondents to this survey are incredibly generous. So a couple of you have asked questions about response rates and what might be a minimal acceptable response rate, and how do we try to maintain the validity of a sample by keeping high response rates. And the way we do it for a long-term panel study like the Wisconsin Longitudinal Study is to try, first of all, we start off, we have extensive efforts to track and locate the members of the sample before we start interviewing. So we make a lot of efforts to try to find people, to make sure we include as many people as we can. We also sent advanced letters before we called them, telling them that the call that they get from the University of Wisconsin is going to be us calling back. 

For a few people, that probably says, oh, no, it's them again, and there's no way I'm answering that phone. But for most of them, it seems to have the effect, oh, yes, it's them, I've invested this much, I might as well go through it again. And they do. They're very cooperative. With the self-administered questionnaire, we include a financial incentive. If people don't want to participate, we put the case aside and we try again later, after a while. It could be they're having a bad day. It could be they're having some financial or health problems and this simply isn't a good time. And we try back again later. There are also special mailings that we sometimes send to reluctant respondents. 

So there's a mailing of music from 1957 in one of the mailings. We sent a special CD of hits of 1957 to try to inspire people to remember some cohort loyalty and motivate them to participate. And we got a few more interviews with that. It was a good investment. But we also, in between the surveys try to maintain contact with the sample by sending out occasional newsletters with results of the survey. There's a special Web site for people that participate in the survey. And last year, there high school reunions, since it was the class of 1957. So researchers were spread out all over the state of Wisconsin, making presentations at high school reunions about the survey, and some of the results from it. This slide, you can't possibly see. I'm just going to tell you what's on here so that you can kind of get the idea. 

This is the original 1957 cohort of 10,000. This is the 1964 postcard. The 1975-77 data collection. The 1992-93 data collection. The 2004-05 data collection. Over here is sort of the summary fact. The response rate was above 85% at each of these waves. So working with a Wisconsin population, and making the kinds of efforts we do to maintain the sample are really key to the value of these data. What kinds of questions do researchers ask with this data that you might not be able to approach so easily in some other ways. Well, some of you may have heard a result. 

And I thank Bob Hauser for passing on some of these findings to me. Some of you may have heard the result that drinking in moderation is associated with improved cognition as people age. Now, that might be very, very welcome news to a number of people. And I know at least one person my age who took up the habit of having wine with dinner every night, which he had avoided before, in response to hearing results like this. It turns out that this is probably, at least using the data analysis in the Wisconsin Longitudinal Study, a statistical artifact. How could that happen? It could happen, for example, and this is what the data analysis suggests. If those who have high cognitive ability in high school, so long before they ever aged, grow up to be moderate drinkers, they also grow up to have high cognitive functioning as they age. So what looks like, as people get older, a relationship between moderate drinking and high cognitive functioning, is simply a result of the fact that people who originally had high cognitive functioning tend to be moderate drinkers, and tend to maintain high cognitive function. Now that, I mean, when I hear something like that, I think, you know, this is the kind of result and insight that you really can only have from a very well-designed panel survey. 

You can't do it very well by simply, as well, by asking people to remember back to the way things were. You can't for example measure past cognitive ability by asking people what was your cognitive ability in high school. Yes? 
- Could that sample have been broken down to those who have high cognitive ability in '57 and who did not become moderate drinkers, say didn't drink at all, and how their cognitive ability ended up? 
- So the question is what about, for example, a sub-group, like those who had high cognitive ability in high school who didn't drink at all. And the question is, sort of what would their outcome be. You can't tell from this analysis, and especially not from this brief summary of it. This is really dealing with averages. But you could with a study like the WLS look at exactly that question. I suspect that you would find that they are also still higher in cognitive ability. 

The implication of this analysis is that the drinking doesn't have any effect on maintaining cognitive ability. It's just that people who start out with high cognitive ability tend to maintain high cognitive ability, on average. But there are always going to be exceptions to it, because it's a statistical analysis that summarizes the experience of a group. Sorry. 
- Couldn't they have broken down the sample into a sub-group of those who tend not to drink moderately? 
- So the question is why couldn't they have broken down the sample into the sub-group of those who did not drink moderately. Say more? Cognitive functioning? 
- And the outcome and what year, 2007 or something, whether they maintained that kind of cognitive ability without the drink. 
- So did they maintain the high level of cognitive ability without the drink? 

They probably could look at that question. I don't know that they looked at it in quite that way. But they might have. But they could. 
- You don't know if they did. 
- I don't. 
- So they probably didn't. 
- I wouldn't say that they probably didn't. I would suspect that they did, because this is a very thorough group of researchers. I was just working from a very schematic summary of the results. 

And in fact, if any of you who wanted to explore the question of, do people with high cognitive ability maintain it regardless of their drinking, the data are publicly available. It's a bit daunting, because there's a lot of it. But anybody here could find access to the data. The other point I wanted to mention that bears on the issue of response rate, is it turns out, and it's not surprising given the level of demand of the Wisconsin Longitudinal Study as I've described it, that those with low cognitive functioning are less likely to participate in the survey. So people who have low cognitive function in high school are less likely to participate in the survey currently. And you can see how, if that's the case, trying to draw conclusions about the relationship between say, cognitive functioning and alcohol use is more complicated if you've excluded part of the range of variation in cognitive ability. So the kinds of efforts that we make to try to get as many respondents as possible are really important for the kinds of scientific questions that researchers are asking. I think in the interest of time, I'm not going to talk about the next example. But I'd be happy to come back to it if you want to in a few minutes. I'll just say and address one more kind of comment about measurement, and show you something about that. 

Then we'll have whatever other questions you have. I mentioned that in addition to threats to validity from sampling, there are also threats to validity from measurement. We talked about two sampling issues: coverage and non-response. The threats to validity from measurement come from having questions that are maybe difficult to present in the same way to everybody, or that respondents have a difficult time answering. And I just wanted to show you two of my favorite answers to questions. I can't say that this is one of my favorite questions. But I like the answers a lot. This is a question that originates from some other survey that's replicated in the Wisconsin Longitudinal Study. And I'll just read it so that you can hear how challenging it might be. Oh, sorry, I don't want to do this one. 

Let's do this one. Without a hearing aid, and while in a group conversation with at least three other people, have you been able to hear what is said? Now, that's a very complicated question. And most interviewers will read it a lot more quickly than I just read it. Nevertheless, it's a question that's clearly designed to get an answer of either yes or no. So without a hearing aid and while in a group conversation with at least three other people, have you been able to hear what is said. This respondent doesn't say yes or no. They say, in this case I think the respondent is male, "Most of the time." Now, for me, that's an extremely interesting answer, because it tells me something about what I would want to fix in this question. Now there are lots of things you might want to fix in this question. 

One of them being the number of conditions that are attached to this, the fact that the person may never have been in a group conversation with at least three other people. There are a lot of things that we could just enjoy complaining about indefinitely. But one thing that's very interesting about the answer is that it points to a different kind of problem with the question, which is that it doesn't make clear what yes is. So do you say yes if you can ever hear in a group conversation with at least three other people, if you can always hear in a group conversation with at least three other people. Or as this respondent volunteered, you can hear in a group conversation with three other people most of the time. So that this respondent seems to be answering the question as though he has this underlying sense of the frequency of being in a group conversation with at least three other people. And if he kind of introspects and looks at that, he would say he could hear most of the time. And that's a very interesting way to answer the question, because it suggests that maybe the question should be something like, how often can you hear in a group conversation with at least three other people. And so for someone like me, who's interested in survey measurement and survey questions, I immediately start making revisions that focus on how often can you hear. But then I meet this respondent. 

And this respondent answers the same question by saying, "Not very good." So that suggests that this person, the previous respondent seemed to be thinking about the frequency with which he could hear. This one seems to be thinking about how well he can hear. Well, that's another possibility. I could ask, how well can you hear in a group conversation with at least three other people. And that probably would have some frequency information in it. But it wouldn't be, probably, only about frequency, but also about the level of hearing. Okay, anybody want to comment on these questions? You probably like them a lot. 
- By mentioning yes or no. 

- The comment is that in each of these cases, the interviewer has influenced the answer by saying yes or no. And in fact, this is a kind of interesting situation in terms of the interviewer's behavior. Because the interviewer is in a kind of funny situation here. Because the respondent has said something that in our normal, everyday frame of mind, we would probably take to mean yes, and in the next case no. It turns out that when interviewers give this kind of leading probe, it's almost always in a situation like this. Why? Well, probably because if somebody says I can hear most of the time, in an everyday way, you would sort of take that to be, well, that's probably yes. And in fact, the formulation I just gave, probably yes. If a respondent says probably yes, interviewers code that as yes. They don't say by probably yes, do you mean, should I put yes or no. 

If a respondent says probably yes, they code it as yes. In a case like this, also, if the interviewer is just very even handed, the respondent ends up feeling like she's not listening. I said most of the time, can't you tell that's closer to yes than it is to no. On the other hand, it really isn't clear what the people who wrote the question wanted. Because you can't tell if they want a yes if it's ever, most of the time or always. And that's a question that if this question were tested using current practices, would probably be corrected. It's the kind of thing that we now find when we test questions. But at the time this question was written, which was some time in the archaeological past of survey research, those methods weren't available. So when interviewers behave in this kind of one-sided way, it's almost always because it sounds in an everyday sense as though the respondent has either indicated yes, as in this case, or no as in this case. And they're sort of trying to respond. 

But the better interviewer behavior would be, well, would you say yes or no, in both cases. 
(inaudible)
 If you ever want to come work for the Survey Center and inspire our interviewers to follow-- 
(inaudible)
 
- Did you interview for the census bureau? 
- For 20 years. 
- You're not Lois, by any chance are you? 
- I am Lois. 
- You interviewed me! 

(laughter)
 My house is in the sample. 
- Really? 
- Yep. You were a good interviewer, too. As her comments might suggest. I actually had two census interviewers named Lois. 
(inaudible)
 I had both of you. 
(laughs)

 It's nice to meet you again after all these years, Lois. What a pleasant surprise. Any other? But yes, she's exactly right. A better practice would also be to repeat the question. And I have some nice examples where the interviewers do repeat the question in these situations. Other comments? Yes. 
- It looks like you can also answer yes if there were 100 conversations and the person could only hear what was said one time. Yes would still be a correct answer. 

- It could be. It could also be that what the designers meant was can you always hear. But they don't say. Right, but it doesn't say "ever," either. It's just plain vague, I think. You're absolutely right. But someone else could also make a slightly different argument about it. 
(inaudible)
 Well, I think at the time-- Why did they put such a vague question in the survey. And the answer is that I think at the time that they wrote it, they thought it was clear. 

And they weren't able to listen to interviews the way we're able to listen to them now to learn that their assumption is wrong. A lot of survey questions are written by investigators sitting at desks. And they never actually, in the past, have actually listened to very many interviews. And you can tell when you read the question. There's a lot of evidence that they've never actually, for example, tried to answer their own question. So your comment is very appropriate. It's puzzling. But it's also understandable when you think about the fact that some of these questions were developed 20 years ago. And at that time, it was not possible to record and listen to interviews the way we can now, with the permission of the respondent. Other questions or comments? 

(inaudible)
 We actually do a hearing check at the beginning of the interview. We make sure the respondent can hear us before they continue. Tom? 
- Since I work at the Biotech Center, I'd be interested what the purpose of collecting the DNA samples from the WLS folks is. 
- I can not. The question is, what's the purpose of collecting the DNA samples from the WLS group, from the sample. What's the purpose of that. I can't address that in any detail. I can address it in a general way, which is to say that there are a lot of health outcomes that are being tracked in the WLS sample. 

We have a cohort that is relatively homogeneous environmentally, socio-culturally. Not totally homogeneous, but relatively homogeneous. So a lot of the variation in their health outcomes will be due to genetic variation. But there also, obviously, going to expect to have some interaction between genetic variation and their health behavior. And the WLS has a long history of health behavior for these people. But what it doesn't have, at least so far, is the genetic piece of information. So the addition of the DNA offers the possibility of a unique data set that combines a really high response rate with information about cognitive ability from an early age, and past socio-economic achievement, as well as health behavior, to try to sort out, help to contribute to discussions about what's the relative contribution of genetic endowment and behavior to health outcomes. Does that answer your question? Sort of? As well as a survey researcher can, rather than say, an investigator. 

- As long as you have a DNA sample, you'll be able to have it for years to come, even after people die, and still go back to that sample and continue to get insights from that sample with technology that might not even be developed now. That's pretty amazing about having all the sociological information being tied to the DNA samples. 
- The comment is that having the DNA available means that not only are we able to do certain kinds of things immediately when we get the DNA and do the assays, but there are technologies that aren't yet available that might make our investigating other research questions possible in the future. And that's right. We're also looking at taking, there's a way that's just been developed, where you can ask people for a blood sample, that they can administer to themselves that's apparently almost painless, and that provides a tremendous amount of other kinds of data that is being considered for a future wave of the Wisconsin Longitudinal Survey. And that means respondents will be able to say they've given their blood. Tom, did you have a finish to that? 
- I just wanted to get that sampling for my kids, because they take 10ml from kids now, and that's a lot to take from a three-year-old. 
- Yeah, but it's not nearly that much. All we want is a drop. 

And we can, you know, we can see the world in a drop of blood. Not me, but other people. Beatrice, and then next. 
- It gets really complicated. We know about DNA. And now you're relating to risky behavior of the respondent. But we also know that there's an intrauterine environment that affects fetal development. And so, when we need to know about the risky behavior of the mother while she was pregnant. 
- So the comment is, if we really wanted to fill out the picture, we need to know something about the intrauterine environment and the risky behavior of the mother. And I think that's probably true. 

And I don't know what kinds of information, completely, that the WLS has about parents. I don't think they have that. They do have information about socio-economic characteristics and job information and so forth for the parents. But probably not that level of detail. Yes? 
- Is the survey lab in the same position as the state cartographic lab, in that you cannot undertake anything that would be in competition with private companies doing this kind of work? 
- So the question is, is the survey lab in the position as the cartographic lab, in that we can't compete with the private sector. And the answer is yes. And we're not often. That doesn't often come up for us, because we're working primarily on grants for NIA, or from the various agencies and institutes within the National Institutes of Health, or for the State of Wisconsin. 

And none of our data can be proprietary. So we really, and our clients tend to be, compared to the clients in the private sector, our clients tend to be, I wouldn't say the poorest of the poor. But they're not generally in a position to buy the services from the private sector. And the kinds of things we do are not available in the same way easily in the private sector. Polling firms, for example, can't do the kind of work we do. They just don't have the infrastructure. They don't train their interviewers the same way. They don't have the same view of what they're trying to do. It's just a different enterprise. It's not less valid or less useful. 

It's just different. And it's not, you can't just switch off one to the other. There were a couple of hands I thought saw in the back. Did the questions go away? Or maybe people were just stretching. Okay, let me just come to a close in kind of a quick way. I wanted to show the logo for the Wisconsin Longitudinal Study, just because you don't often in sociology have a poodle skirt as anything that's relevant, much less a T-Bird. So there's that. I'll just also mention that the University of Wisconsin Survey Center does the Badger Poll, which every once and a while you'll see turn up in the news. I'm not going to talk about that in any detail because of time. 

I wanted to go back and show you, just mention some of the Web sites that materials that I talked about tonight could be found at. You can link to some of them through the University of Wisconsin Survey Center Web site: uwsc.wisc.edu the Centers for Disease Control for BRFSS at a national level, and the DHFS Web site for BRFSS at a state level. And where is my thank you slide? Don't tell me it isn't, there it is. There it is. This is what we've been using as a wall paper in our 20th anniversary mailings, listing some of our principal studies. And there it says thank you. So thank you very much. 
(applause)