⚖️ The News Scale (which measures how balanced the news media are)

I'm all thumbs, but I can now build houses, I wrote a couple of weeks ago. Here is one of the houses I knocked together over the summer. But it needs your help to stand steady.

Unbalanced news media

My criticism of the news media is not that they report negative news. It is the massive tilt toward the negative. That they are so unbalanced between negative and positive news.

The problem is that their imbalance makes us all a little stupid. We get a skewed picture of reality.

Steven Pinker writes about this in Enlightenment Now and published an excerpt from it in this article in The Guardian:

"People estimate the probability of an event or the frequency of a kind of thing by the ease with which instances come to mind."

We think things are worse than they are, and we feel and act accordingly. Pinker again:

"Plane crashes always make the news, but car crashes, which kill far more people, almost never do. Not surprisingly, many people have a fear of flying, but almost no one has a fear of driving."

This affects us, of course. How we feel, how we see the future, and what decisions we make.

If the news media showed a balanced picture, we would on the whole make better decisions about our own future and the future of our world.

That is why Warp News exists. We want to help improve the overall balance by highlighting important and real news that is positive.

But how unbalanced are the news media? I have written about it several times, including about Washington Post and about how much more negative headlines have become over the past twenty years.

The News Scale

Wouldn't it be interesting to see this in real time? To see how (un)balanced the news media are right now, today.

AI has made it possible. Since generative AI is good at understanding and assessing text in this way, and since you can build tools with the help of AI, I have built a news scale.

Every day for three weeks it has evaluated headlines and leads from 26 news outlets in 15 countries, more than 7,600 headlines in total.

First I compiled the research in the field. What methods exist for assessing whether an article is positive or negative? We then used those in the build.

One insight was that both the selection of news and the tone matter. A news item can be neutral but framed negatively (or positively, but that basically never happens). Or it can be negative but described neutrally (which actually does happen).

To make it comparable with established research, we chose a five-point scale, from −2 (very negative) to +2 (very positive). Selection and tone are reported separately but are also added together into a total scale.

Other insights gave the News Scale the following design:

Blind assessment. Language models have proven to classify news texts more accurately and consistently than hired human raters, but they are influenced by the name of the source. So the AI sees only the headline and lead, never which news outlet the text comes from.

Three independent rounds. Each headline is assessed three times, in randomized order, by calls that do not know each other's answers. The score is the average; the spread is reported as consistency, in practice 85–90 percent full agreement.

News journalism only. Editorials, columns, opinion pieces, sports, culture, and lifestyle are excluded.

Latest measurement

The latest measurement was carried out, at the time of writing, yesterday, August 31.

66 percent of the headlines were rated as negative, 22 percent as neutral and 11 percent as positive. That gives an imbalance of -55.

The most negative yesterday were NPR in the US and Der Spiegel in Germany, with 100 percent negative news. The only one with more positive than negative was Graphic Online from Ghana.

Limitations

Even though 26 different outlets is a respectable number, the selection is heavily skewed. American, Western European, and Swedish news outlets clearly dominate.

The measurement is taken once a day, around 4 pm CET, and captures what is visible at that moment. Ideally it would run several times a day, or better still, be a live assessment.

It is only headlines and leads. Ideally you would want an assessment of the full article text.

The reason is that this analysis is fairly expensive. One run currently costs about $4. That is not much in itself, but with several hundred outlets, in far more countries, and reading full articles, it would cost considerably more.

I will try to find funding to at least expand the number of news outlets and cover more countries.

Your help is needed!

One way to both check and improve the quality of the measurement is to compare the AI's assessments with those of humans. So I have added a feature on the website where you can rate five news headlines the same way the AI does.

The human results are then compared with the AI's. That way we can see whether they agree, and if not, what we can do to improve the AI's assessment.

It is a simple beta version, which over time (and with your help) I hope to make both bigger and better. My hope is that if news outlets can see, again and again, how unbalanced they are, they will become more aware and perhaps slowly begin to change.

Here you can check out the beta page for The News Scale, and at the top there is a button that lets you rate headlines.

Of course, I welcome tips and ideas on how the site and the measurement can be developed. Please also report any bugs or other errors you find.

Mathias Sundin
Angry Optimist