As scientists, it can be easy to lock ourselves inside the “science bubble” and ignore how the general public perceives our work. While some researchers may see the importance of communicating findings to the public, others may argue that “research is for researchers” or even that psychological research is too complicated to communicate to the public. Despite the variety of opinions on the matter, the decision is ultimately up to members of the public: pay attention to research if one finds it interesting and valuable or ignore the research if not. The question is, then, how does the public receive psychological research? Indeed, past researchers have asked similar questions: is psychological research relevant (Giner-sorolla et al., 2018; Lilienfeld, 2012; Silverman, 1971)? Are psychologists asking the kinds of questions that are important and interesting to non-psychologists (Sherif, 1970)?
One way to examine the level of interest in psychological research is to examine how people engage with and share research on social media websites. Reddit is a free social media website with a large worldwide audience (500+ million subscribers). Reddit is the second-most used site for news and the 18th most-trafficked website (Alexa, n.d.) and is considered to be one of the most influential websites on the internet (Shaer, 2012). With such a large audience, this website provides an excellent window into online trends and the interests of the public.
One subsection of this website (called a “subreddit”) is called r/Science, which allows users to engage with the most recent scientific research and media coverage. All users are subscribed by default to this section. Anyone can post content, comment, “upvote” content they like, and “downvote” content they do not like. Further, posts are tagged by the poster as belonging to various scientific disciplines. However, the title of the post is often not the title of the published paper nor is the journal name required to be advertised. Further, posts to r/Science, must be peer-reviewed research (no summaries, secondhand reviews, pop articles, etc), the research must be less than 6 months old, and the subreddit is heavily moderated to enforce these rules. Thus, such a metric is a straightforward method of assessing whether psychological research is interesting to the average user and, qualitatively, what kinds of research topics are of most interest.
For the purposes of this investigation, data were downloaded from Reddit which include posts to r/Science from all of 2017 (data were downloaded in mid 2018, so 2017 was the most recent complete year). The graph below (Figure 1) depicts the mean upvotes per discipline. Posts tagged with “Psychology” have the highest mean upvotes (M = 2,759, SE = 302.95) followed closely by those tagged with “Social Science” (M = 2,602, SE = 447.34) and “Health” (M = 2,525, SE = 253.35). Clearly, Redditors are interested in research which deals with humans and psychology is a clear example of this.
Breaking down the data even further, I had four research assistants categorize each psychology post into a broad sub-discipline (social, developmental, clinical, or neuroscience) based on the topic of the research as indicated by title of the post. The coders disagreed on 28% of the titles; for these 252 titles, an independent coder reviewed the disagreements and served as the tie-breaker. Figure 2 shows the distributions of posts according to the four categories. Those categorized as social psychology were the most in number followed by clinical, developmental, and finally neuroscience, which received fewer than 1,000 upvotes.
The next goal was to examine the “Social Psychology” posts more closely. However, the post itself does not share the same name as the article, nor does it always link directly to the article (it may link to a press release). By following the link from the post on r/Science and searching the press releases, the journal in which each article was published was identified. This allowed for the journal-level metrics to be recorded, including H-index, SJR score, impact factor, and SciMago rankings (SCImago, n.d.). In addition, the number of citations from Google Scholar and the Altmetric score for each article were also recorded at the request of reviewers.
The first thing to notice about these data are that the posts with the highest number of upvotes came from a variety of journals (see Table 1), few of which are considered the highest-ranking journals in social psychology. Looking at the top 10 most upvoted posts offers a glimpse into the kinds of research Redditors find most interesting. One thing to notice about these data is that the top 10 most upvoted posts received between 37,000 and 74,000 upvotes: psychological research is hardly going unnoticed.
|74,636||Journal of Applied Research in Memory and Cognition||Taylor, Wood, & Garry, (2017)||In a new study, Americans disproportionately chose the years of their own youth as the countrys greatest years no matter how old they were now. This finding is the latest involving a phenomenon known as the reminiscence bump.|
|57,898||Journal of Experimental Social Psychology; Perspectives on Psychological Science||Frimer, Skitka, & Motyl, (2017); Ditto et al., (2017)||Two new psychology studies show that liberals are just as prone to confirmation bias as conservatives.|
|54,090||Frontiers in Psychology||Hoehl, Hellmer, Johansson & Gredebäck, (2017)||Snakes and spiders evoke fear and disgust in many people. There has been debate about whether this aversion is innate or learnt. Scientists discovered that babies as young as six months old feel stressed when seeing these creatures, long before they could have learnt this reaction.|
|52,571||Sex Roles||Robinson, Anderson & White, (2017)||Thanks to a decline in homophobia, straight men are now better able to embrace “bromances” with other men – which the study describes as being “more emotionally intimate, physically demonstrative, and based upon unrivaled trust and cohesion compared to their other friendships”.|
|48,250||Journal of Experimental Social Psychology||Cowgill, Rios, & Simpson, (2017)||Study: Atheists behave more fairly toward Christians than Christians behave toward atheists.|
|46,692||Personality and Individual Differences||Weiser et al., (2017)||Study of 550 college students who had used or were using Tinder finds about one in five participants admit to talking with a person on the dating app while in an exclusive romantic relationship.|
|44,505||Consciousness and Cognition||LaFrance & Cuttler, (2017)||A new study suggests that marijuana users are more creative, but it’s probably not because of their marijuana use.|
|43,125||Social Psychological and Personality Science||Zitek & Jordan, (2017)||People with a greater sense of entitlement are less likely to follow instructions than less entitled people are, because they view the instructions as an unfair imposition on them, finds new research in the journal Social Psychological and Personality Science.|
|39,140||Proceedings of the Royal Society B: Biological Sciences||Brienza & Grossman, (2017)||A study has found that people with the lowest social class scores, those with less income, less education, and more worries about money scored about twice as high on the wise reasoning scale as those in the highest social class.|
|37,583||Personality and Individual Differences||Ståhl & van Prooijen, (2018)||A person is more effective at analyzing fake news and conspiracy theories if they have a tendency for analytical thinking, which provides consistent protection against conspiratorial thinking and other irrational beliefs, but only if it was accompanied by a belief in the value of critical thinking.|
While some of the most upvoted posts may belong to “top tier” journals, other higher-ranking journals frequently had their articles posted on Reddit even if these were not the most upvoted posts. Looking at the journals which had the most posts (see Figure 3) we see that journal status does not necessarily indicate public interest. The Journal of Experimental Social Psychology (IF = 2.87) had some of the most upvoted articles while Psychological Science (IF = 6.13) and Journal of Personality and Social Psychology (IF = 5.73), for example, had more posts with lower mean upvotes within a tighter range.
Examining this data at the journal level changes the information we can glean. Not only are we asking what kinds of psychological research Reddit users find interesting, but we can now see whether the questions of interest and relevance to this audience are in any way related to metrics used by social psychologists. That is, are metrics like journal ranking and impact factor a good indicator of what research receives upvotes? It appears they are not.
For all posts categorized as belonging to social psychology, correlations between Reddit upvotes and the journal-level and article-level metrics were examined. Given the skew of upvotes, I also examined the natural-log of upvotes. In either case, Reddit upvotes were simply not correlated with impact factor or journal rankings (see Table 2). Please note that the n varies across correlations for each metric because not all metrics are available for all journals. For example, not all journals in the dataset have a ranking in the SciMago “social psychology” category and not all journals have published impact factors (SCImago, n.d.).
|2||Reddit Upvotes (natural log)||.66*|
|3||Journal Impact Factor||–.06||–.06|
|4||ournal Rank: Overall||.01||–.04||–.46**|
|5||Journal Rank: Social Psyc.||–.05||–.06||–.57**||.99**|
|6||Journal Rank: Psychology||.08||.04||–.66**||.99**||1.00*|
|8||Journal SJR Score||–.06||–.06||.85**||–.54**||–.70**||–.73**||–.23**|
|9||Journal cites/3 years||.01||.00||–.05||.03||–.68**||.00||–.08||–.11*|
|11||Article Altmetric Score||.03||.10*||.17**||–.16**||–1.0||–.09||–.08||.16*||.06||.07|
Metrics such as impact factor and journal rankings are often factored into decisions for hiring, tenure promotions, and grants (but, see Abambres, Ribeira, Sousa, & Lantsoght, 2018, for an argument as to why impact factors are poor metrics of researcher performance). Thus, these results further suggest that the kind of research that makes it into top journals is not necessarily the same research that is upvoted on Reddit.
Importantly, this data contains information about many articles from the same journal, so it’s also important to check for interdependence. Thus, I attempted to construct a multi-level model to identify what proportion of variance in upvotes can be accounted for by the journal. However, there was almost no variance at the journal-level (ICC = .005), which means that the reliability of measurements within each journal are highly inconsistent. This suggests that a mixed model accounting for both article and journal level variance is not needed to understand the patterns in these data.
What do these results mean? First, these data show that Redditors care deeply about research which deals with humans—namely psychological research, and especially social psychological research: many posts were upvoted tens of thousands of times. For comparison, the other most upvoted r/Science categories were Social Science, Health, Engineering, and Environment. The first, tenth, and twentieth number of upvotes are displayed in Table 3. As you can see, social psychology is not too far ahead nor too far behind these other categories. Thus, one take-home message is that Redditors care about psychological research.
|Position||Social Psychology||Social Science||Health||Engineering||Environment|
However, it’s also worth noting that posts in r/Science are still far from being the most upvoted posts on the entire Reddit website. For example, the most upvoted post of all time on Reddit received 349,000 upvotes (Reddit, n.d.) and Reddit posts from other subreddits frequently exceed 100,000 upvotes (Pereira, 2018).
The reasons why some articles are shared on Reddit and others are not is not a question that can be answered definitively with the present data. However, examining the ten most upvoted articles in Social Psychology (Table 1) suggests that users of r/Science appreciate descriptive research and research that offers up practical findings which describe something interesting about ourselves—for example, politics, religion, dating, personality, identity, etc.
In contrast, one can also speculate about the kinds of research which people find less interesting. For example, past research suggests that over time psychology has become more complicated, esoteric, and long-winded (Witte & Brandt, 2011), as well as overly theoretical and less descriptive (Rozin, 2001), and increasingly complex (Quiñones-Vidal, José López-García, Peñaranda-Ortega, & Tortosa-Gil, 2004; Sherman, Buddie, Dragan, End, & Finney, 1999). Compared to the descriptive and practical findings noted in Table 1, this combination of characteristics seems less likely to attract the average reader even though it may gain praise from scientists (Reis & Stiller, 1992).
Second, these data point to a disconnect between the metrics of evaluation used by scientists and general audiences. These data show that the research people find interesting on Reddit is not necessarily associated with scientific publishing metrics. Other research has reported that the public attention an article receives (e.g. Altmetric score) is not a good predictor of its future citation count (Thelwall & Nevill, 2018) which further suggests a disconnect between the metrics used by science and readers. There are a few possible explanations for this disconnect which can be explored in future research (but are not testable with the present data). For example, one could speculate that scientists may be concerned with theory and lay people may be concerned with application. Perhaps scientists want something to build their new findings on and lay people want findings that apply in the present moment. Indeed, the articles shown in Table 1 are more practical rather than theoretical. Another possibility is that maybe scientists see a different goal and a different audience for their science. Again, these possibilities are speculative but could be examined in further experimental studies. Whatever the reason, I think these data suggest that people find at least some psychological research interesting and perhaps “relevant” despite concerns otherwise.
Of course, the data presented here are not meant to be prescriptive; that is, I would not recommend that scientists go on Reddit to identify the areas that lay people find interesting and start conducting their research accordingly. Perhaps this would be a great way to decide on the topic for a popular science book, instead. However, the pursuit of flashy and surprising findings is one factor leading to large-scale non-replicability (Collaboration, 2015; Camerer et al., 2018), possibly because the qualities of risky and surprising findings differ from the kinds of research yielding expected (or “boring”) results (Fiedler, 2017).
Instead of planning out research based on what will be “popular,” perhaps we can reflect on who should be the beneficiaries of our science and whether our science is having the desired effect. As psychology is the field aiming to explain human behaviors (Rad, Martingano, & Ginges, 2018), should we be producing findings that can apply to people’s life in the present? From this data, it appears that psychology as a whole does fulfill that role (to some extent) whether scientists like it or not. Perhaps scientists should be trying to identify the areas of research which are most impactful and beneficial in everyday life. Research identifying these areas of study would be most welcome.
Therefore, these findings have some important implications for issues around the replicability and integrity of our research. People will pay attention to our findings, they will share them online, they will talk about them, and they will use them in their lives. We (scientists) then have a responsibility to make sure that our published findings are accurate and hold up to scrutiny. We have a responsibility to make sure that we share data, we are transparent in our analyses, and that we minimize the reporting of false-positives. We have a responsibility to make sure that we are communicating our results accurately because whether we like it or not, our research does not just go to researchers.
The take-home message is that there both is and isn’t a “crisis of relevance” (Giner-Sorolla, 2019). In one sense, there may be a crisis because the kinds of research that lay people find interesting is not necessarily the kind of research that gets cited or published in high-impact journals; this disconnect may be concerning. However, people (at least users of Reddit) clearly find some psychological research interesting enough if it is tied for the highest mean upvotes on r/Science. I hope that we, psychologists, can continue to garner interest and that we can continue to ask the right questions. At the same time, I hope the field can continue to improve the methods we use and the way we report our findings because people are watching eagerly.
Data described herein can be accessed at: https://osf.io/bca4v/?view_only=6e3a3f379c1f4a86bbdec04fbe69746d.
Special thanks to Viet Duong and Phu Pham who accessed the data for me; special thanks to Thuy-vy Nguyen who provided comments on the drafts of this manuscript.
The author has no competing interests to declare.
Abambres, M., Ribeiro, T., Sousa, A., & Lantsoght, E. (2018). Research Counts, Not the Journal. OSF Preprints. DOI: https://doi.org/10.2139/ssrn.3368683
Brienza, J. P., & Grossmann, I. (2017). Social class and wise reasoning about interpersonal conflicts across regions, persons and situations. Proceedings of the Royal Society B: Biological Sciences, 284(1869). DOI: https://doi.org/10.1098/rspb.2017.1870
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., Altmejd, A., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637. DOI: https://doi.org/10.1038/s41562-018-0399-z
Cowgill, C. M., Rios, K., & Simpson, A. (2017). Generous heathens? Reputational concerns and atheists’ behavior toward Christians in economic games. Journal of Experimental Social Psychology, 73, 169–179. DOI: https://doi.org/10.1016/j.jesp.2017.06.015
Ditto, P. H., Liu, B. S., Clark, C. J., Wojcik, S. P., Chen, E. E., Grady, R. H., & Zinger, J. F., et al. (2018). At least bias is bipartisan: A meta-analytic comparison of partisan bias in liberals and conservatives. Perspectives on Psychological Science. DOI: https://doi.org/10.1177/1745691617746796
Frimer, J. A., Skitka, L. J., & Motyl, M. (2017). Liberals and conservatives are similarly motivated to avoid exposure to one another’s opinions. Journal of Experimental Social Psychology, 72, 1–12. DOI: https://doi.org/10.1016/j.jesp.2017.04.003
Giner-Sorolla, R. (2019). From crisis of evidence to a “crisis” of relevance? Incentive-based answers for social psychology’s perennial relevance worries. European Review of Social Psychology, 30(1), 1–38. DOI: https://doi.org/10.31234/osf.io/nua9s
Hoehl, S., Hellmer, K., Johansson, M., & Gredebäck, G. (2017). Itsy bitsy spider…: infants react with increased arousal to spiders and snakes. Frontiers in psychology, 8, 1710. DOI: https://doi.org/10.3389/fpsyg.2017.01710
LaFrance, E. M., & Cuttler, C. (2017). Inspired by Mary Jane? Mechanisms underlying enhanced creativity in cannabis users. Consciousness and cognition, 56, 68–76. DOI: https://doi.org/10.1016/j.concog.2017.10.009
Lilienfeld, S. O. (2012). Public skepticism of psychology: Why many people perceive the study of human behavior as unscientific. The American Psychologist, 67, 111–129. DOI: https://doi.org/10.1037/a0023963
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). DOI: https://doi.org/10.1126/science.aac4716
Pereira, A. (2018). These were the most upvoted Reddit posts this year. Retrieved May 30th, 2019 from www.SFchronicle.com
Quiñones-Vidal, E., Loźpez-García, J. J., Peñaranda-Ortega, M., & Tortosa-Gil, F. (2004). The nature of social and personality psychology as reflected in JPSP, 1965–2000. Journal of Personality and Social Psychology, 86(3), 435. DOI: https://doi.org/10.1037/0022-35220.127.116.115
Rad, M. S., Martingano, A. J., & Ginges, J. (2018). Toward a psychology of Homo sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences, 115(45), 11401–11405. DOI: https://doi.org/10.1073/pnas.1721165115
Reddit (n.d.). The senate. Upvote this so that people see it when they Google “The Senate.” Retrieved May 20th, 2019 from: https://www.reddit.com/r/movies/comments/62sjuh/the_senate_upvote_this_so_that_people_see_it_when/?st=JIPKDMZC&sh=6fcba263
Robinson, S., Anderson, E., & White, A. (2018). The bromance: Undergraduate male friendships and the expansion of contemporary homosocial boundaries. Sex Roles, 78(1–2), 94–106. DOI: https://doi.org/10.1007/s11199-017-0768-5
Rozin, P. (2001). Social psychology and science: Some lessons from Solomon Asch. Personality and Social Psychology Review, 5(1), 2–14. DOI: https://doi.org/10.1207/S15327957PSPR0501_1
SCImago (n.d.). SJR — SCImago Journal & Country Rank [Portal]. Retrieved April 20th, 2019 from http://www.scimagojr.com
Shaer, M. (2012, July 8). “Reddit in the Flesh.” New York Magazine. Retrieved from http://nymag.com/news/features/Reddit-2012-7/
Sherif, M. (1970). On the relevance of social psychology. American Psychologist, 25, 144–156. DOI: https://doi.org/10.1037/h0029429
Silverman, I. (1971). Crisis in social psychology: The relevance of relevance. American Psychologist, 26, 583. DOI: https://doi.org/10.1037/h0031445
Ståhl, T., & Van Prooijen, J. W. (2018). Epistemic rationality: Skepticism toward unfounded beliefs requires sufficient cognitive ability and motivation to be rational. Personality and Individual Differences, 122, 155–163. DOI: https://doi.org/10.1016/j.paid.2017.10.026
Taylor, R. J., Burton-Wood, C. G., & Garry, M. (2017). America was great when nationally relevant events occurred and when Americans were young. Journal of Applied Research in Memory and Cognition, 6(4), 425–433. DOI: https://doi.org/10.1016/j.jarmac.2017.05.003
Weiser, D. A., Niehuis, S., Flora, J., Punyanunt-Carter, N. M., Arias, V. S., & Baird, R. H. (2018). Swiping right: Sociosexuality, intentions to engage in infidelity, and infidelity experiences on Tinder. Personality and Individual Differences, 133, 29–33. DOI: https://doi.org/10.1016/j.paid.2017.10.025
Witte, E. H., & Brandt, V. (2011). Social psychological research: The comparison of four journals. Retrieved from http://psydok.sulb.uni-saarland.de/volltexte/2011/2743
Zitek, E. M., & Jordan, A. H. (2017). Psychological Entitlement Predicts Failure to Follow Instructions. Social Psychological and Personality Science. DOI: https://doi.org/10.1177/1948550617729885
The author(s) of this paper chose the Open Review option, and the peer review comments are available at: http://doi.org/10.1525/collabra.223.pr