How should we investigate variation in the relation between social media and well-being?
DOI:
https://doi.org/10.15626/MP.2022.3322Keywords:
Social media, well-being, effect heterogeneityAbstract
Most researchers studying the relation between social media use and well-being find small to no associations, yet policymakers and public stakeholders keep asking for more evidence. One way the field is reacting is by inspecting the variation around average relations—with the goal of describing individual social media users. Here, we argue that this approach produces findings that are not as informative as they could be. Our analysis begins by describing how the field got to this point. Then, we explain the problems with the current approach of studying variation and how it loses sight of one of the most important goals of a quantitative social science: generalizing from a sample to a population. We propose a principled approach to quantify, interpret, and explain variation in average relations by: (1) conducting model comparisons, (2) defining a region of practical equivalence and testing the theoretical distribution of relations against that region, (3) defining a smallest effect size of interest and comparing it against the theoretical distribution. We close with recommendations to either study moderators as systematic factors that explain variation or to commit to a person-specific approach and conduct N=1 studies and qualitative research.
Metrics
References
Aalbers, G., Vanden Abeele, M. M. P., Hendrickson, A. T., de Marez, L., & Keijsers, L. (2021). Caught in the moment: Are there person-specific associations between momentary procrastination and passively measured smartphone use? Mobile Media & Communication, 2050157921993896. https://doi.org/10.1177/2050157921993896
Anvari, F., Kievit, R., Lakens, D., Przybylski, A. K., Tiokhin, L., Wiernik, B. M., & Orben, A. (2021). Evaluating the practical relevance of observed effect sizes in psychological research. PsyArXiv. https://doi.org/10.31234/osf.io/g3vtr
Anvari, F., & Lakens, D. (2021). Using anchor-based methods to determine the smallest effect size of interest. Journal of Experimental Social Psychology, 96, 104159. https://doi.org/10.1016/j.jesp.2021.104159
Appel, M., Marker, C., & Gnambs, T. (2020). Are social media ruining our lives? A review of meta-analytic evidence. Review of General Psychology, 24(1), 60–74. https://doi.org/10.1177/1089268019880891
Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100(3), 603–617. https://doi.org/10.1348/000712608X377117
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001
Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
Bayer, J. B., Triu, P., & Ellison, N. B. (2020). Social media elements, ecologies, and effects. Annual Review of Psychology, 71(1), 471–497. https://doi.org/10.1146/annurev-psych-010419-050944
BBC. (2021). China cuts children’s online gaming to one hour. BBC News. https://www.bbc.com/news/technology-58384457
Beard, K. W. (2005). Internet addiction: A review of current assessment techniques and potential assessment questions. CyberPsychology & Behavior, 8(1), 7–14. https://doi.org/10.1089/cpb.2005.8.7
Beltz, A. M., Wright, A. G. C., Sprague, B. N., & Molenaar, P. C. M. (2016). Bridging the nomothetic and idiographic approaches to the analysis of clinical data. Assessment, 23(4), 447–458. https://doi.org/10.1177/1073191116648209
Bergman, L. R., & Vargha, A. (2013). Matching method to problem: A developmental science perspective. European Journal of Developmental Psychology, 10(1), 9–28. https://doi.org/10.1080/17405629.2012.732920
Beyens, I., Pouwels, J. L., van Driel, I. I., Keijsers, L., & Valkenburg, P. M. (2020). The effect of social media on well-being differs from adolescent to adolescent. Scientific Reports, 10(1), 10763. https://doi.org/10.1038/s41598-020-67727-7
Bolger, N., Zee, K. S., Rossignac-Milon, M., & Hassin, R. R. (2019). Causal processes in psychology are heterogeneous. Journal of Experimental Psychology: General, 148(4), 601–618. https://doi.org/10.1037/xge0000558
Bolker, B. M., Brooks, M. E., Clark, C. J., Geange, S. W., Poulsen, J. R., Stevens, M. H. H., & White, J. S. S. (2009). Generalized linear mixed models: A practical guide for ecology and evolution. Trends in Ecology and Evolution, 24(3), 127–135. https://doi.org/10.1016/j.tree.2008.10.008
Bruggeman, H., Van Hiel, A., Van Hal, G., & Van Dongen, S. (2019). Does the use of digital media affect psychological well-being? An empirical test among children aged 9 to 12. Computers in Human Behavior, 101, 104–113. https://doi.org/10.1016/j.chb.2019.07.015
Bryan, C. J., Tipton, E., & Yeager, D. S. (2021). Behavioural science is unlikely to change the world without a heterogeneity revolution. Nature Human Behaviour, 5(8), 980–989. https://doi.org/10.1038/s41562-021-01143-3
Carr, N. (2011). The shallows—how the internet is changing the way we think, read and remember. Atlantic Books.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.
Council on Communications and Media. (2016). Media and young minds. Pediatrics, 138(5). https://doi.org/10.1542/peds.2016-2591
Coyne, S. M., Rogers, A. A., Zurcher, J. D., Stockdale, L., & Booth, M. (2019). Does time spent using social media impact mental health? An eight year longitudinal study. Computers in Human Behavior, 106160. https://doi.org/10.1016/j.chb.2019.106160
Dahly, D. (2021). Perils of personalized treatment effects. https://statsepi.substack.com/p/perils-of-personalized-treatments
Davidson, B. I., Ellis, D. A., Stachl, C., Taylor, P. J., & Joinson, A. N. (2022). Measurement practices exacerbate the generalizability crisis: Novel digital measures can help. Behavioral and Brain Sciences, 45. https://doi.org/10.1017/S0140525X21000534
DeBruine, L. M., & Barr, D. J. (2021). Understanding mixed-effects models through data simulation. Advances in Methods and Practices in Psychological Science, 4(1), 1–15. https://doi.org/10.1177/2515245920965119
Dickson, K., Richardson, M., Kwan, I., Macdowall, W., Burchett, H., Stansfield, C., & Thomas, J. (2019). Screen-based activities and children and young people’s mental health and psychosocial wellbeing: A systematic map of reviews. EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London.
Dienes, Z. (2019). How do I know what my theory predicts? Advances in Methods and Practices in Psychological Science, 2(4), 364–377. https://doi.org/10.1177/2515245919876960
Dienlin, T., & Johannes, N. (2020). The impact of digital technology use on adolescent well-being. Dialogues in Clinical Neuroscience, 22(2), 135–142. https://doi.org/10.31887/DCNS.2020.22.2/tdienlin
Dienlin, T., Masur, P. K., & Trepte, S. (2017). Reinforcement or displacement? The reciprocity of FTF, IM, and SNS communication and their effects on loneliness and life satisfaction. Journal of Computer-Mediated Communication, 22(2), 71–87. https://doi.org/10.1111/jcc4.12183
Efron, B., & Morris, C. (1977). Stein’s paradox in statistics. Scientific American, 236(5), 119–127.
Ellis, D. A. (2019). Are smartphones really that bad? Improving the psychological measurement of technology-related behaviors. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2019.03.006
Eronen, M. I., & Bringmann, L. F. (2021). The theory crisis in psychology: How to move forward. Perspectives on Psychological Science, 16(4), 779–788. https://doi.org/10.1177/1745691620970586
Fisher, A. J., Medaglia, J. D., & Jeronimus, B. F. (2018). Lack of group-to-individual generalizability is a threat to human subjects research. Proceedings of the National Academy of Sciences, 115(27), E6106–E6115. https://doi.org/10.1073/pnas.1711978115
Frankenhuis, W., & Nettle, D. (2018). Open science is liberating and can foster creativity. Perspectives on Psychological Science, 13(4), 439–447. https://doi.org/10.1177/1745691618767878
Fried, E. I. (2020). Lack of theory building and testing impedes progress in the factor and network literature. Psychological Inquiry, 31(4), 271–288. https://doi.org/10.1080/1047840X.2020.1853461
Gelman, A. (2015). The connection between varying treatment effects and the crisis of unreplicable research: A Bayesian perspective. Journal of Management, 41(2), 632–643. https://doi.org/10.1073/pnas.1711978115
Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, 348(1-17), 3.
Griffioen, N., van Rooij, M., Lichtwarck-Aschoff, A., & Granic, I. (2020). Toward improved methods in social media research. Technology, Mind, and Behavior, 1(1). https://doi.org/10.1037/tmb0000005
Griffioen, N., Scholten, H., Lichtwarck-Aschoff, A., van Rooij, M., & Granic, I. (2021). Everyone does it—differently: A window into emerging adults’ smartphone use. Humanities and Social Sciences Communications, 8(1), 1–11. https://doi.org/10.1057/s41599-021-00863-1
Grimes, T., Anderson, J. A., & Bergen, L. (2008). Media violence and aggression: Science and ideology.
Haaf, J. M., & Rouder, J. N. (2019). Some do and some don’t? Accounting for variability of individual difference structures. Psychonomic Bulletin & Review, 26(3), 772–789. https://doi.org/10.3758/s13423-018-1522-x
Hall, J. A., Xing, C., Ross, E. M., & Johnson, R. M. (2019). Experimentally manipulating social media abstinence: Results of a four-week diary study. Media Psychology, 0(0), 1–17. https://doi.org/10.1080/15213269.2019.1688171
Hamaker, E. L. (2012). Why researchers should think "within-person": A paradigmatic rationale. In M. R. Mehl & T. S. Conner (Eds.), Handbook of research methods for studying daily life (pp. 43–61). Guilford Press.
Hernán, M. A. (2018). The c-word: Scientific euphemisms do not improve causal inference from observational data. American Journal of Public Health, 108(5), 616–619. https://doi.org/10.2105/AJPH.2018.304337
Hester, N., Xie, S. Y., & Hehman, E. (2021). Little between-region and between-country variance when people form impressions of others. Psychological Science, 1–11. https://doi.org/10.1177/09567976211019950
Houghton, S., Lawrence, D., Hunter, S. C., Rosenberg, M., Zadow, C., Wood, L., & Shilton, T. (2018). Reciprocal relationships between trajectories of depressive symptoms and screen media use during adolescence. Journal of Youth and Adolescence, 47(11), 2453–2467. https://doi.org/10.1007/s10964-018-0901-y
Howard, M. C., & Hoffman, M. E. (2018). Variable-centered, person-centered, and person-specific approaches: Where theory meets the method. Organizational Research Methods, 21(4), 846–876. https://doi.org/10.1177/1094428117744021
IJzerman, H., Lewis, N. A., Przybylski, A. K., Weinstein, N., DeBruine, L., Ritchie, S. J., Vazire, S., Forscher, P. S., Morey, R. D., Ivory, J. D., & Anvari, F. (2020). Use caution when applying behavioural science to policy. Nature Human Behaviour, 1–3. https://doi.org/10.1038/s41562-020-00990-w
Johannes, N., Dienlin, T., Bakhshi, H., & Przybylski, A. K. (2022). No effect of different types of media on well-being. Scientific Reports, 12(61), 1–13. https://doi.org/10.1038/s41598-021-03218-7
Kardefelt-Winther, D., Heeren, A., Schimmenti, A., van Rooij, A., Maurage, P., Carras, M., Edman, J., Blaszczynski, A., Khazaal, Y., & Billieux, J. (2017). How can we conceptualize behavioural addiction without pathologizing common behaviours? Addiction, 112(10), 1709–1715. https://doi.org/10.1111/add.13763
Kaye, L. K., Orben, A., Ellis, D. A., Hunter, S. C., & Houghton, S. (2020). The conceptual and methodological mayhem of “screen-time”. Open Science Framework. https://doi.org/10.31219/osf.io/u4hqn
Kross, E., Verduyn, P., Sheppes, G., Costello, C. K., Jonides, J., & Ybarra, O. (2021). Social media and well-being: Pitfalls, progress, and next steps. Trends in Cognitive Sciences, 25(1), 55–66. https://doi.org/10.1016/j.tics.2020.10.005
Kruschke, J. K. (2014). Doing Bayesian data analysis: A tutorial introduction with R (2nd ed.). Academic Press.
Kunert, R. (2016). Internal conceptual replications do not increase independent replication success. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-016-1030-9
Kushlev, K., & Leitao, M. R. (2020). The effects of smartphones on well-being: Theoretical integration and research agenda. Current Opinion in Psychology, 36, 77–82. https://doi.org/10.1016/j.copsyc.2020.05.001
Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. https://doi.org/10.1177/2515245918770963
Liew, S. X., Howe, P. D. L., & Little, D. R. (2016). The appropriacy of averaging in the study of context effects. Psychonomic Bulletin & Review, 23(5), 1639–1646. https://doi.org/10.3758/s13423-016-1032-7
Loh, K. K., & Kanai, R. (2015). How has the internet reshaped human cognition? The Neuroscientist. https://doi.org/10.1177/1073858415595005
Masur, P. K. (2021). Digital communication effects on loneliness and life satisfaction. In Oxford research encyclopedia of communication. Oxford University Press. https://doi.org/10.1093/acrefore/9780190228613.013.1129
Matias, J. N., Pennington, E., & Chan, Z. T. (2022). Testing concerns about technology’s behavioral impacts with N-of-one trials. https://osf.io/jtdng/
Meier, A., Domahidi, E., & Günter, E. (in press). Computer-mediated communication and mental health: A computational scoping review of an interdisciplinary field. In S. Yates & R. E. Rice (Eds.), The Oxford handbook of digital technology and society. Oxford University Press.
Meier, A., & Reinecke, L. (2020). Computer-mediated communication, social media, and mental health: A conceptual and empirical meta-review. Communication Research, 1–28. https://doi.org/10.1177/0093650220958224
Mitev, K., Weinstein, N., Karabeliova, S., Nguyen, T., Law, W., & Przybylski, A. (2021). Social media use only helps, and does not harm, daily interactions and well-being. Technology, Mind, and Behavior. https://doi.org/10.1037/tmb0000033
Molenaar, P. C. M. (2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement: Interdisciplinary Research & Perspective, 2(4), 201–218. https://doi.org/10.1207/s15366359mea0204_1
Molenaar, P. C. M. (2013). On the necessity to use person-specific data analysis approaches in psychology. European Journal of Developmental Psychology, 10(1), 29–39. https://doi.org/10.1080/17405629.2012.747435
Molenaar, P. C. M., & Campbell, C. G. (2009). The new person-specific paradigm in psychology. Current Directions in Psychological Science, 18(2), 112–117. https://doi.org/10.1111/j.1467-8721.2009.01619.x
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie, N., Simonsohn, U., & Wagenmakers, E.-J. (2017). A manifesto for reproducible science. Human Nature Behavior, 1, 1–9. https://doi.org/10.1038/s41562-016-0021
Muthukrishna, M., & Henrich, J. (2019). A problem in theory. Nature Human Behaviour. https://doi.org/10.1038/s41562-018-0522-1
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114
Odgers, C. L., & Jensen, M. R. (2020). Annual research review: Adolescent mental health in the digital age: Facts, fears, and future directions. Journal of Child Psychology and Psychiatry, 61(3), 336–348. https://doi.org/10.1111/jcpp.13190
Orben, A. (2020a). The Sisyphean cycle of technology panics. Perspectives on Psychological Science, 15(5), 1143–1157. https://doi.org/10.1177/1745691620919372
Orben, A. (2020b). Teenagers, screens and social media: A narrative review of reviews and key studies. Social Psychiatry and Psychiatric Epidemiology. https://doi.org/10.1007/s00127-019-01825-4
Orben, A., Dienlin, T., & Przybylski, A. K. (2019). Social media’s enduring effect on adolescent life satisfaction. Proceedings of the National Academy of Sciences, 116(21), 10226–10228. https://doi.org/10.1073/pnas.1902058116
Orben, A., & Przybylski, A. K. (2019). Screens, teens, and psychological well-being: Evidence from three time-use-diary studies. Psychological Science, 30(5), 682–696. https://doi.org/10.1177/0956797619830329
Orben, A., Weinstein, N., & Przybylski, A. K. (2020). Only holistic and iterative change will fix digital technology research. Psychological Inquiry, 31(3), 235–241. https://doi.org/10.1080/1047840X.2020.1820221
Parry, D. A., Davidson, B. I., Sewall, C. J. R., Fisher, J. T., Mieczkowski, H., & Quintana, D. S. (2021). A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use. Nature Human Behaviour, 1–13. https://doi.org/10.1038/s41562-021-01117-5
Parry, D. A., Fisher, J. T., Mieczkowski, H., Sewall, C. J. R., & Davidson, B. I. (2021). Social media and well-being: A methodological perspective. Current Opinion in Psychology. https://doi.org/10.1016/j.copsyc.2021.11.005
Przybylski, A. K., Nguyen, T. T., Law, W., & Weinstein, N. (2021). Does taking a short break from social media have a positive effect on well-being? Evidence from three preregistered field experiments. Journal of Technology in Behavioral Science. https://doi.org/10.1007/s41347-020-00189-w
Przybylski, A. K., & Weinstein, N. (2017). A large-scale test of the Goldilocks hypothesis: Quantifying the relations between digital-screen use and the mental well-being of adolescents. Psychological Science, 28(2), 204–215. https://doi.org/10.1177/0956797616678438
Richters, J. E. (2021). Incredible utility: The lost causes and causal debris of psychological science. Basic and Applied Social Psychology, 43(6), 366–405. https://doi.org/10.1080/01973533.2021.1979003
Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1(1), 27–42. https://doi.org/10.1177/2515245917745629
Rohrer, J. M., Hünermund, P., Arslan, R. C., & Elson, M. (2021). That’s a lot to PROCESS! Pitfalls of popular path models. PsyArXiv. https://doi.org/10.31234/osf.io/paeb7
Rohrer, J. M., & Murayama, K. (2021). These are not the effects you are looking for: Causality and the within-/between-person distinction in longitudinal data analysis. PsyArXiv. https://doi.org/10.31234/osf.io/tg4vj
Rose, L. T., Rouhani, P., & Fischer, K. W. (2013). The science of the individual. Mind, Brain, and Education, 7(3), 152–158. https://doi.org/10.1111/mbe.12021
Satchell, L. P., Fido, D., Harper, C. A., Shaw, H., Davidson, B., Ellis, D. A., Hart, C. M., Jalil, R., Bartoli, A. J., Kaye, L. K., Lancaster, G. L. J., & Pavetich, M. (2021). Development of an offline-friend addiction questionnaire (O-FAQ): Are most people really social addicts? Behavior Research Methods, 53(3), 1097–1106. https://doi.org/10.3758/s13428-020-01462-9
Schemer, C., Masur, P. K., Geiß, S., Müller, P., & Schäfer, S. (2020). The impact of internet and social media use on well-being: A longitudinal analysis of adolescents across nine years. Journal of Computer-Mediated Communication. https://doi.org/10.1093/jcmc/zmaa014
Senn, S. (2016). Mastering variation: Variance components and personalised medicine. Statistics in Medicine, 35(7), 966–977. https://doi.org/10.1002/sim.6739
Senn, S. (2018). Statistical pitfalls of personalized medicine. Nature, 563(7733), 619–621. https://doi.org/10.1038/d41586-018-07535-2
Shaw, H., Ellis, D. A., Geyer, K., Davidson, B. I., Ziegler, F. V., & Smith, A. (2020). Quantifying smartphone “use”: Choice of measurement impacts relationships between “usage” and health. Technology, Mind, and Behavior, 1(2). https://doi.org/10.1037/tmb0000022
Siebers, T., Beyens, I., Pouwels, J. L., & Valkenburg, P. M. (2021). Social media and distraction: An experience sampling study among adolescents. Media Psychology, 1–24. https://doi.org/10.1080/15213269.2021.1959350
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Thorisdottir, I. E., Sigurvinsdottir, R., Asgeirsdottir, B. B., Allegrante, J. P., & Sigfusdottir, I. D. (2019). Active and passive social media use and symptoms of anxiety and depressed mood among Icelandic adolescents. Cyberpsychology, Behavior, and Social Networking. https://doi.org/10.1089/cyber.2019.0079
Turkle, S. (2012). Alone together: Why we expect more from technology and less from each other. Basic Books.
Valkenburg, P. M. (2022). Social media use and well-being: What we know and what we need to know. Current Opinion in Psychology, 45, 101294. https://doi.org/10.1016/j.copsyc.2021.12.006
Valkenburg, P. M., Meier, A., & Beyens, I. (2022). Social media use and its impact on adolescent mental health: An umbrella review of the evidence. Current Opinion in Psychology, 44, 58–68. https://doi.org/10.1016/j.copsyc.2021.08.017
Valkenburg, P. M., & Peter, J. (2013). The differential susceptibility to media effects model. Journal of Communication, 63(2), 221–243. https://doi.org/10.1111/jcom.12024
Valkenburg, P. M., Pouwels, J. L., Beyens, I., van Driel, I. I., & Keijsers, L. (2021). Adolescents’ social media experiences and their self-esteem: A person-specific susceptibility perspective. Technology, Mind, and Behavior. https://doi.org/10.1037/tmb0000037
vanden Abeele, M. M. P. (2020). Digital wellbeing as a dynamic construct. Communication Theory. https://doi.org/10.1093/ct/qtaa024
VanderWeele, T. J., Jackson, J. W., & Li, S. (2016). Causal inference and longitudinal data: A case study of religion and mental health. Social Psychiatry and Psychiatric Epidemiology, 51(11), 1457–1466. https://doi.org/10.1007/s00127-016-1281-9
von Eye, A. (2009). Universals and individuals—is this the end of the discussion? Measurement: Interdisciplinary Research & Perspective, 7(1), 3–7. https://doi.org/10.1080/15366360802715320
Whitlock, J., & Masur, P. K. (2019). Disentangling the association of screen time with developmental outcomes and well-being: Problems, challenges, and opportunities. JAMA Pediatrics. https://doi.org/10.1001/jamapediatrics.2019.3191
Yarkoni, T. (2020). The generalizability crisis. Behavioral and Brain Sciences, 1–37. https://doi.org/10.1017/S0140525X20001685
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Niklas Johannes, Philipp K. Masur, Matti Vuorre, Andrew K. Przybylski
This work is licensed under a Creative Commons Attribution 4.0 International License.
