[UPDATE: Charts and dataset are being updated automatically and are available for download, links below.]

Web scraping news stories reveals interesting trends about how the media cover the 2020 election campaigns of Donald Trump and Joe Biden.  I used Import.io to access and analyze more than 100,000 news stories from the websites of 1,500 US news organizations in order to compare the media coverage of Trump and Biden in the run up to the 2020 US presidential election.

Continue reading Trump v Biden 2020 election: Web scraping for media sentiment

I like to use web data to answer any question. And questions about COVID-19 are no different.  Over the course of the year, I felt that I had noticed the tone of US corporate communications change from “emergency!” in the early stages of the pandemic to “this is the new normal” and I wondered if this change in tone could be detected in the wider US news media.

I used Import.io to collect 665,000 news stories from the websites of 6,800 US news sources, where the words: “covid”, “coronavirus” or “pandemic” appeared in the headline.  I calculated a sentiment score for each news story and then plotted that against the total number of COVID-19 news stories and the number of US COVID-19 deaths to see what relationships would be revealed.

The most notable result was that while media sentiment towards the pandemic started off negative (it’s a global pandemic after all) it quickly moved to be more neutral, even as reported US COVID-19 deaths peaked and then stabilized around 5,000 per week.

Continue reading Web scraping shows how media coverage of COVID-19 turned “neutral” even as deaths increased