Charlotte Vaughn says research debriefings are an overlooked public engagement opportunity

Hi, thanks so much for having me.

Sure. Great question, and it will be important to understanding the rest of our conversation today. I am a behavioral scientist. I’m a linguist and cognitive scientist by background, and the way that we get our data in those sciences is through experiments or other kinds of interactions with people. We’re trying to understand how the mind or the brain works, so we do behavioral experiments with humans—human subjects research. Those typically have three components in an experimental session. If you ever took a psychology class in undergrad and had to do some sort of experimental requirement as part of that class, you could be thinking about that here. Basically, you start out with informed consent. That’s really important. We need to make sure participants know what they’re getting into, that they know they can leave at any time, and that sort of thing. Once the participant has consented, if they do, then you move into the experiment itself. In my world, that could be listening to people talking and making judgments about it, reading something, or it could involve fMRI or EEG. You’re doing the experimental task, and debriefings come at the end. The idea is that this is an opportunity to share with participants what the study was about. I can come to this later or sooner if you want, but debriefings were introduced because many studies were using deception as part of the experimental design. There are some cases where it would not be helpful for your research question if the participant knew the true purpose of the study ahead of time, so you sort of fudge it a little bit. Then the debriefing is a chance to come back and say, “Well, actually, we told you the study was about this. We had to do that for this reason, but what it was really about was this.” It also gives participants a chance to ask questions or withdraw their consent at that point if they’ve changed their minds.

Actually, no. Debriefings are only federally mandated, at least in the U.S., in cases involving deception. If you have any kind of deception in your study, yes, you have to have a debriefing. These are regulated through university ethics boards called IRBs, or institutional review boards. Otherwise, the extent to which debriefings are adopted in studies that don’t involve deception varies widely.

I think one of the things we need is a good landscape of the field and an understanding of the norms around debriefing, and that’s a project I would love to pursue. I can give you my impressions from working in the field for 15 years or more and having colleagues elsewhere. The guidance is pretty fractured across institutions. In the case of deception, your institutional IRB is going to ask to see a debriefing, and they’re going to give you guidance about that. If your study doesn’t involve deception, some IRBs want to review a debriefing. Other IRBs say, “Why are you showing this to me? I don’t really care.” Some departments have subject pools with undergraduates who participate in studies. They want to look at the debriefings, and they might have some guidance. Sometimes there is a template with the name of the study, the goal of the study, why the participant was recruited, and those kinds of things. I don’t actually know how often that happens because there aren’t good data on this. When studies were more commonly done in the lab, you might have a psychology or neuroscience lab on campus and invite undergraduates or community members to come in. The debriefing was often a sheet of paper that you handed to the participant at the end and said, “Thank you for participating. Bye.” Now that many studies have moved to online platforms, the debriefing tends to be a simple screen of text at the end that usually gives the researcher’s specific goals for the study.

That, I would say, is the main variable. If it is in person, the way labs usually work in the behavioral sciences is that undergraduate students are the ones running participants. They’re the ones who schedule participants and bring them back to the lab. Sometimes there will be what’s called a debriefing script, and instead of just handing them the paper, the student might read what’s on it and then hand it to them. There are also times when you might want to know, for your own information about the study’s results, whether participants caught on to the condition they were put into. The conversation is typically centered on the researcher’s goals: “I’m trying to understand what you thought the study was about and what you guessed about our intent.” There have been pushes in psychology, especially for institutions that have subject pools, to make debriefings educational for participants as well. People have been writing about that for a while. The entire rationale for the existence of subject pools—the idea that an undergraduate can get partial credit for a psychology course by doing experiments outside of class—is that it teaches the undergraduates, the participants, something about the research process, science, and the scientific method. There have been a lot of questions about how true that actually is. Do they get much out of participation, and can we make debriefings more of a tool for that? That’s the most variation I’ve seen in how debriefings work in traditional settings, like the lab and online. We can also talk about other kinds of settings.

In public engagement and science communication, we have the traditional touchpoints with science that members of the public might have in mind. There are informal science settings like museums, zoos, and aquariums. We have popular science books, social media accounts, and YouTube channels. Scientists can give public lectures, participate in science cafés, or go to schools. Those are all really important, and I’m also engaged in some of those activities. But I think actual participation in research is an overlooked public touchpoint with science. If we think about an individual behavioral researcher doing however many experiments a year, that single researcher might be interacting, in some limited capacity, with thousands of members of the public per year. As researchers, I’ve noticed that even if we really care about public engagement, there’s this divide: when I’m doing research, I’m doing research; when I’m doing public engagement, I’m doing public engagement. Yet when we’re doing public engagement, we might have to seek out audiences. We’re always trying to find new audiences, while meanwhile we have this stream of people either coming into our labs or doing our research studies online. We’re not really thinking about the fact that those are members of the public who might be encountering our field for the first and only time during that research study. I think it’s an overlooked opportunity for scientists to ask, “What do I want to do with that opportunity? I have this chance to potentially connect with a member of the public.”

Sure. I alluded to this a little bit. A lot of my work has been in traditional formats, like in-person studies in the lab and online studies. But I also have a longstanding interest in public engagement, and I have a lab set up at a museum in Washington, D.C., called Planet Word. The lab is called the Language Science Station, and we do research with visitors to the museum as our participants. The museum is about words and language. Our research is about words and language. It’s behavioral or neuroimaging research. When you’re in that public setting, it’s obvious that we need to make participant-visitors’ experiences in our research studies something that doesn’t detract from their visit to the museum, but is actually seen as adding value. We want it to be fun, educational, and interactive. We put a huge amount of time and effort into making the studies fun and gamified. The museum is very interactive, and we wanted to keep that spirit. We also wanted participants to get something out of their experience, so we did that through debriefings and extended conversations with participants. I can tell you more about that if you’re interested. I had these two versions of my research life: I was putting all this attention on participant experience and what people were getting out of it in the museum, but I was really on autopilot in the lab and online. Then came a moment of thinking, “Wait a minute. We can take more of what we’ve learned from the museum setting and adapt it for our studies in the lab and online.” If we scaled that up, and other researchers thought about it in that way too, we could have a much bigger impact because the huge majority of research is done in the lab and online. It was a mindset shift for me to realize, “I’ve been ignoring this huge space. How can we bring what we’ve learned from the museum and reconceptualize how we think about debriefings in more traditional settings?”

The goal is to reach as many interested behavioral researchers as possible, as well as anyone who works with human subjects. There are biomedical researchers doing clinical trials and other work where I think this is also applicable. In casual conversations with colleagues who don’t work in the museum setting, and with colleagues I’ve worked with at the museum, there’s this understanding that once it hits you—“Oh, right, we could be doing this better”—the next response is, “But how? Also, I don’t have time.” The last thing I want to do is give researchers yet another hurdle to jump through to get their studies off the ground. But because some version of a debriefing is already part of researchers’ pipelines, this is a way to reach people who are curious about public engagement or science communication but feel they don’t have the time or training. We can learn about some principles of science communication and public engagement and apply them to the debriefing setting. I’m excited for some people to take this and run with it and do all kinds of cool things, but I also want to ask whether there are simple tweaks we can make to increase the chances that our debriefings will have the impacts we want. The main goal was to develop a resource toolkit. I have a guidance document that gives some principles of science communication and lays out this thinking about how we can view research as an opportunity for engagement. I also have a detailed worksheet researchers can use with either an existing debriefing they want to revise or a brand-new study. It helps them identify their main message, the right format, and their goals. There’s a lot that I’ve learned through the museum, but there’s also much more work to do to determine what makes an effective, impactful debriefing.

Absolutely. That’s the word I use a lot.

I’ve developed the resources almost completely, or at least the current draft. One of the things about this project is that, as I mentioned, there’s not a lot of data on what guidance is out there or what the landscape is. One outcome I’m hoping for is that people find my resources, find me, and get in touch to say, “This is how it works at my university,” or, “This is what we’re up against,” and brainstorm how to make this as easy an entry point as possible. I have the guidance document and worksheet ready. I’m also compiling a bibliography of the work that’s out there. It does exist—not a ton, especially compared with other parts of the research process like informed consent, which has a huge literature—but I want to make it available too, because researchers tend to like reading research about what they’re working on. I’ve given one version of the online workshop, and I’m looking forward to giving more. A lot of the workshop involves setting the stage and then giving people a chance to apply some of these ideas to their own studies, workshop them, and get feedback.

Yes. The target audience is absolutely the behavioral scientists running the studies. Again, I hope to expand to biomedical scientists and other people working with human subjects. I think there are a couple of different audiences. Graduate students are a big one. A lot of graduate students start running their own studies, while by the time you’re the PI of a lab, you’re usually pretty far removed from actual interactions with research participants. One thing I would strongly suggest is that I learned so much by being at the museum and actually interacting with museum visitors as participants. I hadn’t done much of that for quite a few years because my research assistants were running many of the participants. Getting back to talking with participants about the research itself is very energizing. It helps the research, I think, and it’s also an opportunity for public engagement. People anywhere in the academic pipeline could be an audience—anyone who is interacting with human subjects, basically.

One of the comments I received in the recent workshop was from a faculty member who had done some work at museums. They had struggled with, “We want this to be educational in some way. We don’t really know how.” They worked on it through trial and error, and they got there. Honestly, that’s a big part of how we work in the lab at the museum, the Language Science Station, too. We do a lot of piloting—trying things out and seeing how they land. But she said, “If we had had any kind of guidance, it would have been much easier.” That was my experience: we’ve been doing this for a number of years and for more than 12 studies at the museum. I’ve been doing it with other studies online as well, fleshing out how we could reconceptualize debriefing and what it could look like. It seemed like I couldn’t be the only person thinking about this, but I didn’t see any guidance out there. It’s not necessarily, “Here’s the exact template you should use,” or, “Here’s what you should do.” It’s really asking people to think differently. A big part of it is the goals and objectives you have for your debriefing. If we think about Besley and Dudo’s strategic science communication, it’s basically taking that and applying it to the debriefing. That’s a big part of the groundwork for the workshop and worksheet.

There are some things we’re learning about how to make a better debriefing. That’s one set of things. A lot of what I’m learning is what there still is to do. I’m ready to give some guidance, and then it’s like, “Well, I don’t actually have an empirical basis for a lot of this guidance.” There need to be more studies. We need to do good evaluation and impact assessment on debriefings, just as we would for a wholesale public engagement or science communication project. A lot of that starts with goals and objectives. One of the main things I’ve learned from working with scientists is that once we’re on board and want a debriefing to have more impact on participants, the question becomes: what do you want to do? It’s the same thing Besley and Dudo hear: “I need to teach them something. I want to transmit knowledge. That is the objective of this debriefing.” That can be a great objective, and it makes a lot of sense in the context of debriefing, but it’s not the only objective we could have. A big part of what I’m learning is that scientists don’t think about other possible things their debriefings could be doing: sparking curiosity, changing attitudes or behaviors, promoting goodwill and trust in science, or simply making the experience personally meaningful or memorable to the participant. That alone has been one of the biggest shifts I’ve seen. Scientists think, “What I’m allowed to say or do in this debriefing is communicate unidirectionally with the participant about this incremental thing we’re doing in this study.” One of the biggest eye-openers has been that we can have many different goals with our debriefing. We can introduce people to our field and what we care about more generally, the methods we’re using, or ideas about how science and the process of science work. There are a lot of different angles we can take, and because of the way scientists are trained, that doesn’t tend to enter the possibility space.

That was something else I learned: I need more time during the workshop. The first version was a one-hour workshop, which was trying to cram too much into a short time. I introduce people to the idea, and I’ve divided the workshop into goals and objectives for a debriefing. Once we have the idea that we can do something more with debriefings, the question is, “How would I do that? What does it look like?” First, we concentrate on goals and objectives and broaden the possibility space. Then we move into content, with the content driven by your goals and objectives. It gives researchers permission to say something a little different from only, “This was my incremental research question in this particular study.” Then we move to the form of the debriefing, which should ideally follow from your goals and objectives. Typically, debriefings are solid walls of text, but we play around with what would happen if it were a video or comic strip. I’m starting to do some preliminary work on that. I don’t think having a catchy, cool, interactive form is necessarily the right answer for everyone, and it may not make a debriefing as effective as it could be. It depends on the study, the goals, and the objectives. That’s where I could see science communicators and researchers working together. If I had all the funding in the world, that would be a very cool project.

Great point. The time constraint is real in a couple of different ways. The researcher’s time is the main thing I’ve been thinking about, but there’s also the participant’s time. The conditions under which people do experiments usually involve either an undergraduate coming into a lab, someone recruited to come into a lab for payment, or someone doing something online for payment. Researchers might think, “If I make my debriefing longer, I’m going to have to pay people more, and I don’t have that kind of money.” So there’s that time constraint as well. One thing I’m trying to emphasize, which addresses both time issues, is that this doesn’t mean behavioral scientists have to develop yet another suite of skills in video editing or other areas. You don’t have to make a half-hour-long program after your study. It’s about shifting mindsets and thinking, “If my goal is to spark curiosity, what are some great science communication products that already exist? Are there cool YouTube videos or podcast episodes I can link to?” If your study is conducted online, you’re already in a space where you can say, “If you thought this was cool, look at these other resources you might find interesting.” That won’t require much time from the researcher or the participant. You also asked about repeat participation. Usually, it’s a one-shot thing. You do have participants, especially in online settings, who make extra money as a side gig by doing many research studies. If this became more common, a real question would be: what would that ecosystem look like if we took this seriously and tried to reach participants more as part of the debriefing? We can recognize that researchers have limited time. I would love it if many people thought, “This is very cool,” and it became their gateway into public engagement and science communication. I’m trying to do that a little subliminally as part of the project as well.

For the workshop itself, I created an evaluation using a retrospective pre-post design. I could get into assessment design later, but that’s a separate question. I looked at the goals I have for this particular workshop. A lot of it is about getting people to think differently about what a debriefing can do, especially their goals and objectives, the content of what they’re communicating, and the main message. I had questions like, “Before the workshop, how much did you think about this? After the workshop, how much did you think about this?” That’s what I’m focusing on most for the workshop itself. For the materials, I’m in the process—literally this week—of putting them on the website and implementing tracking: how many people are looking at them, how many are downloading the resources, and whether people get in touch if they find them useful or don’t find them useful. I want to have that dialogue about what would be helpful, what barriers people are encountering, and what other dimensions of the situation we could think about together.

That is absolutely one of the things I would love to do, and I actually have a proposal going to a funder about this. I would love to do more: get cohorts of researchers to work on these things together and in collaboration with science communicators; track what changes they made and the barriers and challenges they encountered; track the impact on researchers; and then also track the impact on participants. As I was saying before, what makes an effective or impactful debriefing? Another pie-in-the-sky area, with more funding, involves a handful of software platforms that behavioral scientists use to design and host experiments online. One group I talked to is really interested. Right now, they estimate that less than 10% of their online studies have debriefings, so we’ve been discussing a way to build a debriefing module into their experiment builder. That’s another way to make this as easy as possible for researchers. With funding, we could do a really cool study at scale to see how researchers implemented it and what the impacts were on participants. We could follow up with participants and researchers later. A platform-wide study would be a really cool thing to do. I also think that funding to bring science communicators together with researchers would be valuable. Researchers would get a lot from learning from science communicators. This is an overgeneralization, but science communication and public engagement tend to focus on hard and applied sciences, with less focus on social and behavioral sciences. I’m hoping it would go in both directions. Science communicators could say, “There’s this whole area that is pretty interesting and that we could be communicating more about.” I’m hoping that might open new opportunities for collaboration too. Again, with an unlimited budget—if we’re talking pie in the sky—I’ll have to go there.

It doesn’t have to be unlimited. Just—

That’s a great question. It is a very walled-off system. Sometimes, even in academia, it’s hard to figure out who is in charge of what stage and what part of the process. A lot of it is about building relationships with researchers and potentially with academic professional organizations. If you’re a science communicator who has a beat in the behavioral sciences, make sure you’re known to the relevant academic organizations. I’m hoping we’ll start to see more interest in this, but I acknowledge that these are very early days in what I hope will be a mindset shift toward getting researchers to think about debriefings as an opportunity for science communication. Down the road, there may be more requests like, “I’m looking for someone who can make art for this,” or, “Help me write a really concise and engaging script for a video or audio piece.” We’re probably a little far away from that. But make those connections with researchers and make yourself known. I’m also hoping we can publicize the relevant work that’s already out there. Behavioral economists, for example, might be looking for a great podcast episode to link to from a debriefing. Being visible—which I know is a goal of science communicators in general—would be fantastic, even just so researchers can link to existing content.

I think there’s a lot of work that still needs to be done. The landscaping, absolutely—just understanding what’s out there in terms of the ethics and the work from science and technology studies, science communication, and all the adjacent fields. We need more work to understand the state of debriefings. This is my own conjecture, but I think the way debriefings were introduced was in reaction to studies that had not done a great job with deception and that had serious ethical violations. Debriefings have been positioned as a defensive tool: “I’ve got to cover my bases. I’ve got to make sure I’m doing deception as ethically as possible.” I want to claim them as proactive opportunities for communication and engagement. One thing I’ve been looking into is that all parts of our research communicate something to participants, whether researchers are aware of it or not: what the task is, what the instructions say, and what stimuli participants hear. Participants are making inferences about science, the scientists, their beliefs, what they care about, and their relationship to participants. Throughout the whole research process, there’s a lot to learn and many studies to be done that interrogate each of those parts. What inferences are people already taking away, and how can I be more intentional in my research? I have a recent paper called “Your Research Is Public Engagement” that tries to make that point and get people to recognize that they can’t opt out of communication as researchers. Your study is going to communicate things. What you can do is be more intentional about what it’s communicating. There are lots of opportunities for the science of science communication to investigate this as a different public touchpoint with science.

There’s a wide-open space here, and we don’t know a lot about it. We should try to better understand the space and fund people who are interested in capitalizing on this opportunity to reach members of the public who are encountering science in this way. There’s a lot we could do for researchers, science communicators, and members of the public. Many science communication funders are more interested in hard and applied sciences, and there aren’t as many opportunities for behavioral scientists to apply for funding for science communication and public engagement projects. I would love to see that opened up because then you get an entire ecosystem of people doing public engagement work, with the potential to do it through the research process itself. It’s a cool, different approach to public engagement.

Well, the studies themselves—if you’re participating on a platform like Prolific, then you can opt in and get paid small amounts of money for small amounts of time. That is an opportunity that exists.

Again, I wish I had more statistics about this. I don’t know, for example, if you signed up to be in 20 studies on Prolific, what percentage would contain a debriefing. I would say it’s not the norm to advertise ahead of time that you will have a debriefing. You would be surprised at the end. But participating in behavioral research is a cool way to learn more about science. Some studies will be interesting and thought-provoking. Some will be very boring. But if debriefings are there and done well, they can be really cool ways to learn more about the science of what makes us human, what makes our minds work, and how we can do all the really cool things we do. If that’s something you’re interested in, I think you would get something out of participating in research studies. Hopefully, most researchers leave opportunities for participants to give feedback to scientists. I would encourage participants and members of the public to take them up on that. As you said, debriefings could be an opportunity for more bidirectional communication. I’ve gotten research ideas from talking to museum visitors who participated in one of my studies. One was a study about memory, and we asked people, “Why do you think you remembered this set of things, but your daughter remembered this other set?” From those conversations, it was like, “Oh, yeah, nobody’s ever tested that. Why don’t we look at it?” You can have a chance to contribute to science and, hopefully, learn some cool things in the process.

Thank you so much. This was a lot of fun, and I’d love to come back. Thanks.