In this Papers Podcast, Dr. Johnny Felt and Dr. Chad Shenk discuss their co-authored JCPP paper ‘Contamination bias in the estimation of child maltreatment causal effects on adolescent internalizing and externalizing behavior problems’ (https://doi.org/10.1111/jcpp.13990).
There is an overview of the paper, methodology, key findings, and implications for practice.
Discussion points include:
- Definition of what is meant by the term ‘maltreatment’.
- What is contamination and why is contamination an issue in the study of child maltreatment?
- Challenges and limitations of the study.
- How contamination has been traditionally addressed in child maltreatment studies and how this study has tried to do things differently.
- The implications of the findings.
- How contamination in child maltreatment research should be addressed in future research.
In this series, we speak to authors of papers published in one of ACAMH’s three journals. These are The Journal of Child Psychology and Psychiatry (JCPP); The Child and Adolescent Mental Health (CAMH) journal; and JCPP Advances.
#ListenLearnLike
Subscribe to ACAMH mental health podcasts on your preferred streaming platform. Just search for ACAMH on; SoundCloud, Spotify, CastBox, Deezer, Google Podcasts, Podcastaddict, JioSaavn, Listen notes, Radio Public, and Radio.com (not available in the EU). Plus we are on Apple Podcasts visit the link or click on the icon, or scan the QR code.
Johnny Felt, Ph.D. is a research assistant professor in the Center for Healthy Aging at Penn State. His research interests involve the identification and application of advanced statistical methods to better understand the short- and long-term consequences of stress. More specifically, he uses both Bayesian and frequentist approaches to analyze cross-sectional and longitudinal data, from observational and experimental designs, to investigate how exposure to stress and early life adversity can affect daily stress processes and biobehavioral developmental and aging processes.
Chad Shenk, PhD, ABPP, is a Professor in the Departments of Human Development & Family Studies and Pediatrics at Penn State. Dr. Shenk’s basic science research is centered on improving methods for causal estimation and target identification in prospective cohort studies of child trauma and adverse health across the lifespan. This work identifies biomarkers and etiological mechanisms of various health conditions in the child trauma population using a multiple timescales (real-time, longitudinal) and multiple levels of analysis (biological, behavioral, environmental) approach. His clinical trials research therefore centers on the optimization of behavioral interventions for child trauma by engaging identified targets and mechanisms more effectively. As Principal and Co-Investigator, Dr. Shenk has secured over $30 million in federal grant awards from the National Institutes of Health, National Science Foundation, and other national funders. He is a Fellow of the American Psychological Association (Divisions 37 and 53) and a board-certified clinical psychologist actively seeing patients exposed to child trauma through Penn State’s Department of Pediatrics.
Transcript
[00:00:10.000] Mark Tebbs: Hello, and welcome to the Papers Podcast series for the Association for Child and Adolescent Mental Health, or ACAMH for short. I’m Mark Tebbs. I have a background in psychology, mental health commissioning, coaching and freelance consulting. In this series, we speak to the authors of papers published in one of ACAMH’s three journals. They are the Journal of Child Psychology and Psychiatry, commonly known as JCPP, the Journal of Child and Adolescent Mental Health, known as CAMH, and JCPP Advances.
If you’re one of the fans of our Papers Podcast series, please subscribe on your preferred streaming platform, let us know how we did, with a rating or review, and do share with friends and colleagues.
Today, I’m delighted to be talking with Dr. Johnny Felt and Dr. Chad Shenk, lead authors of the JCPP journal paper, “Contamination Bias in the Estimation of Child Maltreatment Causal Effects on Adolescent Internalizing and Externalizing Behavior Problems.” Johnny, Chad, lovely to be speaking with you. Let’s start with some introductions. Maybe you could tell us a little bit about yourself, your career to date, and your research interests? Johnny, could you start us off?
[00:01:12.760] Dr. Johnny Felt: My name’s Johnny Felt, and I’m currently an Assistant Research Professor, in the Center for Healthy Aging at Penn State. I received my PhD in Quantitative Psychology and Health Psychology from UC Merced in 2018, and I completed two postdoctoral fellowships at Penn State, the last one, actually, with Chad. So, my research interests are in identifying and applying innovative statistical methods, to better understand how stress and trauma impacts short and long-term health and aging outcomes.
[00:01:38.628] Mark Tebbs: Chad?
[00:01:39.256] Dr. Chad Shenk: Hi, my name’s Chad Shenk, Professor of Human Development and Family Studies, and Professor of Pediatrics at Penn State University. I am also a Clinical Psychologist by training. I received my PhD in Clinical Psychology in 20027. I am also licensed and board certified, so I actually see children and adolescents who are affected by maltreatment, which is an issue that guides both my clinical practice, but also my research and teaching responsibilities at Penn State.
Like Johnny, my research interests are in trying to apply causal inference methods to observational work on paediatric trauma, particularly child maltreatment, so that we can get a more accurate understanding of just how bad this adverse event is, and its contributions to a wide range of health outcomes.
[00:02:25.931] Mark Tebbs: Brilliant, thanks for your introductions. I think you worked with some other colleagues on the paper, so an opportunity to give them a name check, and maybe say what they did on the paper.
[00:02:36.504] Dr. Johnny Felt: Yeah, so we had a lot of different people on this paper. I did want to highlight two grad students that were co-authors on this: Anneke Olson and Ulziimaa Chimed-Ochir, they both contributed a lot to the writing and conceptualisation, and Ulziimaa was very helpful with the analytic approach that we used in this paper. There was also a postdoc, Yanling Li at Penn State, and she was really helpful for helping us handle the missing data analysis, and another Professor at Penn State, this was Professor Zach Fisher, and he was another Methodological Consultant for us.
And I also worked with the University of Maryland Associate Professor, Kenneth Shores, and he was – so, he was an expert in using causal inference methods and observational data, and, finally, was Stanford University, Professor Nilam Ram, he’s another Quantitative Methodologist that really helped, kind of, shape the results of this paper.
[00:03:27.659] Mark Tebbs: Brilliant, thanks for that. Let’s turn to the paper, give us just, like, a brief overview.
[00:03:33.304] Dr. Johnny Felt: So, this paper’s, kind of, part of a larger programme of research, led by Chad Shenk, investigating the impact that contamination in non-maltreated control groups has on causal effect estimates in child maltreatment and behaviour problems. So that’s, kind of, just generally what this paper was about.
[00:03:49.569] Mark Tebbs: There’s some quite technical parts to the paper, aren’t there? So, to begin with, could we get a little bit of a definition about what we mean by the term ‘maltreatment’?
[00:03:56.905] Dr. Chad Shenk: Generally speaking, and this is a definition that applies all across the world, an act of maltreatment is any act of commission or omission on the part of a caregiver towards an individual that’s under the age of 18. And those acts include both physical abuse, sexual abuse, emotional abuse, and neglect.
[00:04:17.388] Mark Tebbs: So, what is contamination? And associated with that question is, why is contamination such an issue in the study of child maltreatment?
[00:04:27.545] Dr. Chad Shenk: So, if we could zoom out just a little bit to orient us through this issue. So, whenever we do an experiment or any, kind of, study that has a control group or a comparison group, there are a number of assumptions that we have to make first, in order to come up with what we think is the right causal estimate or risk estimate for a particular outcome in that study.
One of those assumptions is called the “Stable Unit Treatment Value Assumption,” or SUTVA, for short. What SUTVA tells us is that if you are assigned to a treatment or an exposure condition, that you basically stay in that condition, and if you’re assigned to a control or a comparison condition, that you stay in that condition. Basically, you don’t receive multiple treatments or exposures or different levels of those treatments or exposures within the course of a particular study. And that’s true for randomised trials, it’s true for observational work that doesn’t use randomisation.
So, for us, contamination is an example of a SUTVA violation, and that essentially means that there are a certain number of people, in a control or a comparison condition, who have either received the treatment or have been exposed to the event that’s under scientific investigation. From that perspective, the application to child maltreatment research is pretty straightforward. Contamination in this area of research just essentially means that an unknown proportion of children, adolescents, adults, who are in a comparison condition have, in fact, already been exposed to maltreatment, or will be exposed to maltreatment during the course of study.
The reason why this is such an issue in child maltreatment research is because the methods that we use to establish maltreatment and comparison groups are actually pretty poor. They don’t do a great job capturing all instances of maltreatment, so that we know kids who are in the maltreatment group have certainly been maltreated, and no-one in the comparison group has been maltreated.
So, what we tend to do in this area of research is rely on one method, to create those maltreatment and comparison groups, and not know that there’s X% of people in that comparison group who’s actually been exposed to maltreatment. That’s a problem because it’ll bias our estimates and make it harder to discover effects, and it’ll also reduce the magnitude of those effects.
[00:06:46.948] Mark Tebbs: Okay, so, I’m just wondering, how has contamination traditionally been addressed in child maltreatment studies? And then, thinking a little bit about your study, how has this study tried to do things differently?
[00:07:02.305] Dr. Chad Shenk: I’ll let Johnny speak to a part of that, and just to answer your question pretty quickly, much of the field hasn’t addressed contamination, at all. We just assume that whenever we use a particular measure of maltreatment that our maltreatment and control groups are as they should be, and that that SUTVA assumption holds, but oftentimes it doesn’t.
And so, really the importance of this particular study and this programme of research coming out of my lab is really to highlight how contamination is so prevalent in child maltreatment studies. It really is a feature of the work that we do, and to bring attention to that, and to develop some methods that would allow us to correct that bias, so that we can get closer to truth about just how adverse child maltreatment is for a range of outcomes.
[00:07:47.366] Mark Tebbs: How did you go about your study? Were there any particular challenges or limitations that you had to address that you would be able to share with us?
[00:07:56.632] Dr. Johnny Felt: Yeah, I’ll take that question. So, for this study, we actually use national data that’s already available to the public, called the “Longitudinal Studies of Child Abuse and Neglect,” LONGSCAN for short. In this study, children with and without substantiated cases of child maltreatment, and these were substantiated, like, a government agency, who were followed every two years during childhood and adolescence. We ended up getting measurements of substantiated child maltreatment every two years, and because this was measured every two years, we also were able to get self-reports of child maltreatment at many of those occasions. And those self-reports repeated over time, and retrospective at the end of the study, were what allowed us to, kind of, identify whether contamination occurred in the control groups.
In our study, we specifically used the data from the age four wave of data collection until the age 16 wave of data collection, because that’s when our primary outcome, child behaviour problems, was actually measured. So, one of the methodological challenges was, how do we determine when contamination occurred? So, for this study, it was relying on those self-reports, from the children, of whether they reported maltreatment occurring or not. And then specifically looking in the control group, so, what proportion of children in the control group self-reported maltreatment?
There’s a few other methodological challenges that we had to address, as well, especially with causal inference. So in this study, because we couldn’t randomise children to the condition, we had to use different methods and design features, to provide more evidence that a causal effect did exist and rule out threats to that. So, the study design of LONGSCAN really leads us to rule out many threats, because the longitudinal design, several of the children in the study have assessments of the outcome before maltreatment occurred and after. So we do get an ability to, kind of, look at a time series of the outcome, before and after, to help us understand whether the change actually occurred.
And because of this design, we were actually able to use a new causal inference method, that’s commonly used in the econometrics literature, but hasn’t really been used in the psychological and behavioural sciences yet, called the “Synthetic Control Method.” So, just briefly, this approach allows us to construct a control group that more accurately reflects what would have been observed in the maltreated children, had they never been maltreated. And it does this by leveraging the repeated assessments of the outcome before maltreatment occurred.
So, what we end up with, the control group is now a synthetic child who’s never been maltreated, and that gives us a stronger comparison to rule out several potential threats to the causal inference. But one of the limitations with this is that we needed repeated assessments before maltreatment occurred. And most children who have experienced maltreatment tend to have experienced it by the age of four. So in this study, it’s important to know that these are all children who were first maltreated around age ten and later, so it’s really a sample of children who receive maltreatment for the first time in adolescence.
[00:10:53.293] Dr. Chad Shenk: And just to clarify, Johnny, I think you had mentioned, we used substantiated indicators of maltreatment. We actually used the confirmed designation from the Modified Maltreatment Classification System, a subtle difference.
[00:11:05.412] Dr. Johnny Felt: Yeah, important one.
[00:11:07.777] Mark Tebbs: Okay, so what were your findings? What came out from the work?
[00:11:12.691] Dr. Johnny Felt: Yeah, so the main findings, there’s actually some pretty interesting and glaring, kind of, results here. One thing that we found is that the rate of contamination in the control group, in this sample from the LONGSCAN, for 67% of the children in the non-maltreated control group, so more than half of the control group, said that they did receive maltreatment at some time during childhood and adolescence. Where if we would have just only used that substantiated designation, we would have assumed that nobody in the control group was maltreated.
This is important, because in our paper, we analysed the results as if you typically would and not acknowledging that contamination has occurred, and we found no significant causal relationships between child maltreatment status and behaviour problems, even though behaviour problems are an established finding in the child maltreatment literature. So, if a Researcher was looking at this data, they may conclude that, like, oh, child maltreatment doesn’t cause behaviour problems.
But when we decided to remove the children who self-reported child maltreatment in the control group, we found that we were now able to detect causal effects. And the effect size estimates increased from between 17 and 54% in their magnitude, depending on which – how long since the maltreatment occurred, and whether we were looking at internalising or externalising behaviour problems.
[00:12:37.034] Mark Tebbs: Were you expecting the contamination size to be that large? Was that a surprise to you, as Researchers?
[00:12:44.585] Dr. Chad Shenk: We’ve been using this cohort to examine contamination in a couple of other studies, so we had some idea about what it might be. But this particular cohort is – at least in terms of prevalence, there’s only a handful of studies that we’ve done – the field has done, on this issue of contamination. So, a prevalence rate of 65% is the highest that we’ve been able to publish so far, that does, I think, strike us as being pretty high. I mean, think about just how much of your comparison group that is, and how that can affect results.
Johnny, what were you going to say?
[00:13:17.029] Dr. Johnny Felt: I was going to say the same thing, like, we have found fairly high prevalence in other cohorts, but this one was particularly high.
[00:13:25.643] Dr. Chad Shenk: And in some ways it makes sense, because LONGSCAN’s a multisite study, and some of those children were known to be at greater risk for maltreatment than other children in LONGSCAN. So, we do characterise this particular cohort as being, you know, one that has a fair number of kids who are – either have been maltreated or at high risk for being maltreated. So, that contamination prevalence being higher in the study does make some sense in that light.
[00:13:50.512] Mark Tebbs: Yeah. What’s the implications of these findings? I was particularly struck, previously, you, sort of, said that, you know, contamination hasn’t traditionally always been controlled for in previous studies, so I’m just wondering what the implications are of your findings?
[00:14:04.342] Dr. Johnny Felt: The implications of this can be pretty large, as, in our study, the magnitude of the effect size, when contamination’s ignored, is much smaller, so small that we can’t even detect a significant effect. So, this can have, you know, implications for Researchers looking to develop interventions and prevention efforts. They may say, “Oh, these results are not statistically significant, let’s look at a different outcome,” even though that may be an important outcome.
This can also impact the ability for people to replicate results. So some samples may have lower amounts of contamination, while others have larger amounts, and if you don’t measure it, you don’t really know how much you have. So, we may find fairly large effects in one sample and nothing in another sample. So, that can have downstream effects to, like, meta-analyses, where they may determine that there’s just a lot of variability in the estimates from study-to-study, and, on average, it’s zero. But if you had controlled for the contamination, you may find that there actually is something real there, and that could possibly influence policy later.
[00:15:06.780] Dr. Chad Shenk: I mean, so, the clinical side of me always thinks about this, as well. So, like, as Johnny’s mentioning, if we have, say, causal effects or effect sizes that are smaller than what they really are, that really doesn’t limit what we call ‘statistical power’ for being to detect an effect, or plan a future study. Clinically, if, when I’m doing an interview with a kid who’s saying that they’ve been exposed to maltreatment, I may discount that history if the science is saying there’s no effect for maltreatment, or we can’t find an effect for maltreatment. So, I might look towards other causes or a different – take a different ideological perspective on how to do treatment planning with this particular child and family, that minimises or discounts or doesn’t prioritise the maltreatment history.
[00:15:53.548] Mark Tebbs: Yeah, so it’s really, really critical. So, I’m wondering, how would you like contamination in child maltreatment research to be addressed in future research? Is there anything from your study that suggests a way that research should be carried out going forward?
[00:16:11.144] Dr. Johnny Felt: I think, right now, our best recommendation is to, kind of, take a dual measurement approach, and throughout the course of the study, especially if it’s a longitudinal follow-up study. So, what this means is that if you’re using, like, child substantiations, from, like, a government agency, or from some other established protocol, you should also pair that with maybe a child’s self-report. And that’s because the substantiated cases might miss some cases in the control group, and that self-report will help you get some of those cases that may not have been detected, or met that threshold.
And then if you have repeated assessments throughout the study, you should do both a substantiated measure and the self-report at each wave. And this is important because, you know, by definition, child maltreatment can happen at any time during childhood and adolescence. So, if you start children at age four, some portion of the control group may become maltreated by age ten, so it’s good to measure it along these times. It’s also important to note though that the self-report measures are also not perfect, so you may not get all cases of contamination, but you are taking a step into identifying many of the cases that may be present in your control group.
[00:17:26.314] Mark Tebbs: So, have you got any further follow-up research? Is there anything in the pipeline that you would be able to share with us?
[00:17:34.145] Dr. Chad Shenk: Yeah, so, I think, you know, this issue of contamination and how to both convince people that it exists and to adopt some different methods to be able to account for it and correct it, there’s a programme of research in my lab that we’re currently perusing, there’s two active studies that we have. One is reporting results from a propensity score model that is looking at – it’s a different cohort, but we’re actually able to use caregiver reports in this measure, which are really useful when children, like, very young children, are at risk for, or are exposed to, maltreatment, because infants can’t really reliably report on maltreatment themselves. And it just so happens that maltreatment most often occurs during infancy and early childhood. So, being able to evaluate, you know, the appropriateness of caregiver reports for being able to detect contamination, and allow us to generate causal estimates in a propensity score modelling framework, is an exciting study that we have.
There’s also another project, using propensity score models with a different cohort, where we’re looking at whether or not after controlling for contamination and re-victimisation of maltreatment, so this allows us to learn a little bit more about the timing of maltreatment. So, if maltreatment does occur between birth and age four and we address contamination, how does that causal effect change, if we also address a child’s re-victimisation, so being exposed to maltreatment a second or more times?
[00:19:00.686] Mark Tebbs: Thank you so much. We’re coming to the end of the podcast, is there a final take home message for our listeners?
[00:19:09.064] Dr. Johnny Felt: Yeah, I think the take home I want the listeners to come away with is that in child maltreatment research, contamination, as Chad said, is more of a feature, and it could be very prevalent. And when left unaddressed, can bias the effect size estimates downward, so you’re going to find much smaller effects than may be the truth, and the effects may be biased downward so much that you might not be able to detect a significant effect. So, we would encourage Researchers to, at minimum, try a dual measurement strategy, to try to identify cases of contamination and mitigate this potential problem.
[00:19:48.976] Mark Tebbs: Excellent. Chad, anything you’d like to conclude with?
[00:19:52.038] Dr. Chad Shenk: Yeah, I mean, as Johnny said, I mean, contamination is just baked into the observational research that we do on child maltreatment. That is unfortunate. As somebody who’s been in the child maltreatment research field for long enough in his career, it’s a bit disconcerting to know that not every study that I’ve done on maltreatment has corrected for contamination, but getting the word out that this is a real issue I think is the first step.
The second step, again, as Johnny mentioned, is to build out the number of instruments that you’re using to characterise maltreatment in any given study, that’s going to give you certain advantages for detecting contamination that you won’t get if you just use one method, whether it’s records, whether it’s self-report, or a caregiver report.
So, we’re trying to build up a programme of research that allows us to provide recommendations to Researchers on how to guide their current studies, their future studies, to detect contamination. While, at the same time, developing quantitative methods, like what Johnny did in this paper, that would allow us to not only correct contamination, but generate causal estimates for the effects of maltreatment.
So, if you all would like to find out more information about contamination, and the exciting work that we’re doing in our lab, please follow the link in the chat, for more information about our current projects, and more information about publications, and the current work that we’re doing.
[00:21:13.194] Mark Tebbs: Excellent, thank you so much. It’s been a really interesting conversation, a really important topic. So, for more details on Dr. John Felt and Dr. Chad Shenk, please visit the ACAMH website, www.acamh.org, and follow us on Twitter @ACAMH. ACAMH is spelt A-C-A-M-H, and please don’t forget to follow us on your preferred streaming platform, let us know whether you enjoy the podcast, with a rating or review, and do share with friends and colleagues.