Studying child outcomes of gay and lesbian households is a politicized thing, as I discovered this past summer. Hence it is also a very public thing. Is it not thus reasonable to wonder whether the 78 respondents tracked for the past two and a half decades are truly unaware of — and hence unaffected by — the media attention regularly paid to the study in which they still participate.
The “Hawthorne effect” refers to the tendency of study participants to work harder or perform better because they know they are being studied. While it is typically applied to experimental research studies of worker productivity, the same could be true here. It’s a cousin to “social desirability bias,” which is closer to what I’m suggesting. In this case, I’m concerned that the kids feel pressure to give better-than-accurate portrayals of their household and personal life. When the adolescent children of lesbian parents are being intermittently interviewed for a study whose results have proven quite politically important — and almost always covered favorably by the mainstream media — it’s prudent for scholars to be skeptical about whether respondents are still offering valid and reliable responses years after they were first contacted. Some kids will always offer valid information, but given the fishbowl these 78 have lived in, I’m concerned that social desirability bias will affect disproportionate numbers of them, especially in contrast to far larger survey projects.
To be sure, some information — like high-school graduation or household changes — can seem innocuous to respondents. Other information — about things like emotional health, substance use, and reflections on their childhood — may be more problematic. Being one of these 78 kids is a little like being the child of someone elected to public office. It’s a big responsibility, and people — including the media — are watching you. Could it be that these children’s parents have never sought to influence their responses, or reminded them of the ramifications of their answers, or shown them the media attention accorded them, or simply introduced them to the NLLFS website? The risk of compromised data sources here, I assert, is elevated.
Nevertheless, peer-reviewed journals continue to give the NLLFS data nearly carte blanche, so far as I can tell. And so the media narrative about the study’s children that flows from the NLLFS publicity office and the Williams Institute at UCLA remains one of nonstop good news.
And yet my misgivings are not about the good news. I believe the analysts and authors are not making it up. I just don’t believe the 78 kids in the NLLFS are capable of reporting unbiased information any more, not after a childhood and adolescence spent entirely in a fishbowl. Even the NLLFS’s principal investigators suggested that 25 years of data collection may be enough. I would concur, and — since the study commenced in 1986 — we eclipsed that mark in 2011. Perhaps it’s time to commit significant funds — and a panoply of research perspectives — to a very large (and hence expensive), longitudinal, population-based data-collection effort that would make fans of the NLLFS and fans of the NFSS alike content with its methodology.
I’m all for more information. But if the data are to be valid and reliable, the study needs to be as free of source bias as is humanly possible. I won’t hold my breath, though, because in the case of lesbian parenting, a nationally representative sample is not what many of my scholarly, rational, and allegedly dispassionate colleagues in the social sciences appear to want.
— Mark Regnerus is an associate professor of sociology at the University of Texas at Austin and a research associate with the university’s Population Research Center. He is also the principal investigator of the New Family Structures Study.