Q: How does Ad Fontes Media generate the scores for each news source on the Media Bias Chart?
A: We generate overall news source scores based on scores of individual articles (in the case of online news sources) or episodes (in the cases of podcasts, radio, TV, and video-based sources).
Each individual article and episode is rated by at least three human analysts with balanced right, left, and center self-reported political viewpoints. That is, at least one person who has rated the article self-identifies as being right-leaning, one as center, and one as left-leaning. Articles and episodes are rated in three-person live panels conducted in shifts over Zoom. Analysts first read each article and rate them on their own, then immediately compare scores. If there are discrepancies in the scores, they discuss and adjust scores if necessary. The three analysts’ ratings are averaged to produce the overall article rating. Sometimes articles are rated by larger panels of analysts for various reasons–for example, if there are outlier scores, the article may be rated by more than three analysts.
Q: Do you differentiate between news and opinion articles?
Yes! We rate all types of articles, including those labeled analysis or opinion by the news source. Not all news sources label their content as opinion, and regardless of how it is labeled by the news source, we make our own methodology determinations on whether to classify articles as analysis or opinion on the appropriate places on the chart.
Q: How do you select the articles for each source?
For each news source, we pick a sample of articles that are most prominently featured on that source’s website over several news cycles. We typically have at least 15 articles rated per source, but for our top 100 we have several dozen articles each, and for the largest sources (such as the New York Times and Washington Post) we have over 100 articles each in our sample.
Our content ratings periods for each rated news source are performed over multiple weeks in order to capture sample articles over several news cycles. Sources that have appeared on our Media Bias Chart for longer have articles over much longer periods of time.
Often, our sample sets of articles and shows are pulled from sites on same day, meaning that they were from the same news cycle. Doing so allows analysts to incorporate evaluations of bias by omission and bias by topic selection.
We use a multi-person rating per article system to minimize the impact of any one person’s political bias on the published rating. We purposefully assign each analyst a breadth of coverage over as many sources as possible to to enhance each analyst’s familiarity with sources across the spectrum.
Q: How many sources have you rated?
To date, we have manually rated over 20,000 individual articles and shows. Each article or show is rated by at least three analysts–one left, one center, and one right, politically–which means we have over 60,000 individual ratings. We have fully or partially rated over 1000 individual news sources.
We also rate podcasts and TV shows! So far we have rated over 300 podcasts and 300 individual TV programs that fall under the category of news and “news-like” sources.
Q: How often do you update sources?
We update all sources periodically by adding new articles. Because we have so many news sources, and because the most popular sources are important to the public, we generally update the most popular sources the most frequently and less popular sources less frequently. For example, we update a tier of the top 15 sources with about 15 new articles each month, and the next tier of 15 sources with about 7 articles per month. The top 200 get updated with about 5 articles per quarter and the next 200 about 5 articles per 6 months. We strive to balance rating new sources and updating existing ones.
Q: How, exactly, do your analysts rate articles and episodes?
The type of rating we ask each analyst to provide is an overall coordinate ranking on the chart (i.e., “40, -12”). The rating methodology is rigorous and rule-based. There are many specific factors we take into account for both reliability and bias because there are many measurable indicators of each. The main ones for Reliability are defined metrics we call “Expression,” “Veracity,” and “Headline/Graphic,” and the main ones for Bias are ones we call “Political Position,” “Language,” and “Comparison.” There are several other factors we consider for certain articles. Therefore, the ratings are not simply subjective opinion polling, but rather methodical content analysis. Overall source ratings are composite weighted ratings of the individual article and show scores.
In our current process, we rate most articles during live shifts (on Zoom) with three analysts (one left, one right, one center), and after each article, analysts see each other’s scores and resolve discrepancies when possible. If significant discrepancies remain, the articles are rerated by a second balanced panel.
We continue to refine our methodology as we discover ways to have analysts classify rating factors more consistently. Our analysts use our software platform called CART–Content Analysis Rating system. This ratings software is currently available for use by educators in classrooms, and by individual adult learners in our news literacy courses. Educators and individuals can learn how to rate news articles like Ad Fontes Media. Our courses include detailed video and written explanations of the factors we use to rate articles.
Q: How did you develop this methodology? How did it evolve over time?
Ad Fontes Media’s Founder, Vanessa Otero, originally created the content analysis methodology. This methodology has evolved over time and with input from many thoughtful commentators and experts, including Ad Fontes Media Advisor and long-time journalist and journalism professor Wally Dean. Back in 2016, when Vanessa created the first Media Bias Chart, she performed the analyses alone. However, to improve the methodology, make the process more data-driven, and mitigate bias (hers and that of any new analysts), she recruited teams of politically diverse analysts and trained them in the methodology. Over time, this process evolved into Ad Fontes Media’s current method of multi-analyst content analysis ratings. Ad Fontes finished its first extensive multi-analyst content ratings research project in June 2019,.
From June 2019 to August 2020, a group of nine analysts from that initial project continued to rate several dozen articles per month to add new sources and update previously existing ones. From August-October 2020, Ad Fontes conducted a second large multi-analyst content ratings project to rate over 2000 articles and 100 new news sources with some existing analysts and over 30 new analysts. Prior to October 2020, all analysts were volunteers receiving perks and/or small stipends. Since October 2020, Ad Fontes Media has contracted and hired a team of analysts to rate news content on an ongoing basis. Read our detailed White Paper about our process for more information.
Q: What else should I know about the methodology?
As a background, it is helpful to understand some principles and caveats:
- The main principle of Ad Fontes (which means “to the source” in Latin) is that we analyze content. We look as closely as possible at individual articles, shows, and stories, and analyze what we are looking at: pictures, headlines, and most importantly, sentences and words.
- The overall source rating is a result of a weighted average, algorithmic translation of article raw scores. Low quality and highly biased content weight the overall source down and outward. The exact weighting algorithm is not included here because it is proprietary. Aspects of what is disclosed here are patent pending.
- The current ratings are based on a small sample size from each source. We believe these sample articles and shows are representative of their respective sources, but these rankings will certainly get more accurate as we rate more articles over time.
- Keep in mind that this ratings system currently uses humans with subjective biases to rate things that are created by other humans with subjective biases and place them on an objective scale. That is inherently difficult, but can be done well and in a fair manner. There are other good models for doing something similar, such as grading standardized written exams (like AP tests and the bar exam), or judging athletic competitions (such as gymnastics and figure skating). You can get to good results as long as you have standards on how to judge many granular details, and have experts that are trained on such standards implementing them. We’ve created that process here. Below are some of those granular details.
Our CART rating interface shows each of the factors analysts are asked to consider before providing a final rating for reliability and bias.
As shown, analysts have scoring sliders for each of the reliability sub-factors and bias sub-factors. Each of these sub-factors have specific definitions and criteria, which are available in our training materials and are linked below:
How to Rate Expression
How to Rate Veracity
How to Rate Headlines and Graphics
How to Rate Language
How to Rate Political Position
How to Rate Comparison
Ad Fontes Media analysts start with these training guidelines and receive at least 20 hours of training, which includes live practice rating articles, before rating articles for inclusion in our data set.
Q: Is your underlying data available?
Our full data sets are available for various types of commercial, non-commercial, and educational purposes for a fee. For commercial uses, see our Ad Apex line of products, and for non-commercial and educational uses, see our IMBC Educator Pro product.