Using web data to see how the UK answered Marcus Rashford’s call to “end child food poverty”

[UPDATE: Over the weekend Marcus Rashford continued to use his Twitter account to share announcements from local businesses offering to provide free meals for children over October half term.  As of Monday morning he has shared announcements from over 400+ businesses.]

Over the summer Marcus Rashford, the England and Manchester United football player, led a food poverty campaign so effective that it forced an embarrassing policy u-turn from Boris Johnson.  The England international wrote a moving open-letter to all MPs in Parliament that spoke about his own experience growing up dependent on free school meals and he urged ministers to extend the free school meals scheme, which provides food for 1.3m children during term time, through the summer break while pandemic restrictions remained in place.  The government had not been planning such an extension but facing mounting pressure from the opposition, the media and even their own backbenchers, a last minute change of direction was made and the government agreed to release £120m to fund the scheme.  

This week, schools were out for half-term in England again and on Wednesday there was an Opposition Day Debate in the UK Parliament to discuss the continued provision of free school meals during school holidays for the rest of the year and into 2021.  (The Scottish and Welsh governments have already put in place funding for extensions into 2021 for Scotland and Wales.)  The Conservative government directed their MPs to oppose the extension and so the proposal was defeated by a vote in Parliament on Wednesday.  

Rashford reacted to the defeat by vowing not to give up his campaign.  And so, on Thursday evening he began tweeting

Continue reading Using web data to see how the UK answered Marcus Rashford’s call to “end child food poverty”

Trump v Biden 2020 election: Web scraping for media sentiment

Figure 1. Media sentiment towards Trump & Biden since September 2020

[UPDATE: Charts and dataset are being updated automatically and are available for download, links below.]

Web scraping news stories reveals interesting trends about how the media cover the 2020 election campaigns of Donald Trump and Joe Biden.  I used to access and analyze more than 100,000 news stories from the websites of 1,500 US news organizations in order to compare the media coverage of Trump and Biden in the run up to the 2020 US presidential election.

Continue reading Trump v Biden 2020 election: Web scraping for media sentiment

Web scraping shows how media coverage of COVID-19 turned “neutral” even as deaths increased

I like to use web data to answer any question. And questions about COVID-19 are no different.  Over the course of the year, I felt that I had noticed the tone of US corporate communications change from “emergency!” in the early stages of the pandemic to “this is the new normal” and I wondered if this change in tone could be detected in the wider US news media.

I used to collect 665,000 news stories from the websites of 6,800 US news sources, where the words: “covid”, “coronavirus” or “pandemic” appeared in the headline.  I calculated a sentiment score for each news story and then plotted that against the total number of COVID-19 news stories and the number of US COVID-19 deaths to see what relationships would be revealed.

The most notable result was that while media sentiment towards the pandemic started off negative (it’s a global pandemic after all) it quickly moved to be more neutral, even as reported US COVID-19 deaths peaked and then stabilized around 5,000 per week.

Continue reading Web scraping shows how media coverage of COVID-19 turned “neutral” even as deaths increased

Web scraping is just automated browsing and everyone does it

There is some confusion about ‘scraping’, what it is, whether it is legal and how it can be used.  ‘Scraping’ of the web is just automated access to websites and it is lawful.  Legal departments know this, which is why some of the largest companies in the world use to convert the web into structured data for use in their businesses.

I helped with a blog post that was published yesterday by Jamie Williams from the EFF as part of their Coders’ Rights Project that works to protect programmers and developers, engaged in the cutting-edge exploration of technology, from badly drafted computer crime laws.

Continue reading Web scraping is just automated browsing and everyone does it

Gender differences amongst sex workers online

Two weeks ago the UK’s Office of National Statistics (ONS) announced that the UK economy had grown by 0.9% in the second quarter of 2014. This growth was partly due to the decision to include prostitution in gross domestic product (GDP) for the first time ever.

The ONS estimate that Britain’s 60,879 prostitutes contribute £5.314bn (0.4% GDP) to the UK economy.  But, this figure completely ignores male prostitution.  In a previous post we used data from AdultWork, a popular sex worker website, to determine that 42% of UK prostitutes are male, which (all other factors being equal) represents an additional £3.542bn of UK GDP!

Our initial analysis was very simple, but nonetheless raised a lot of interest from you all, along with some great questions about the gender differences between male and female sex workers.  We decided to look a little deeper into the gender differences amongst sex workers online and this is what we found.

Continue reading Gender differences amongst sex workers online

How much does prostitution contribute to the UK economy?

How many prostitutes are there in the UK?  According to the Office of National Statistics the answer is 60,879.  In figures due to be released next week this number is being used to add £5.314bn to the official size of the UK economy.  But the ONS only attempted to measure the number of female prostitutes.  If male prostitutes are included in the count then the contribution that prostitution makes to the UK economy rises to £8.856bn.

From the 30th of September the UK’s national accounts will attempt to measure illegal transactions to which all parties consent, including the sale of illegal drugs and prostitution.  Illegal transactions are difficult to measure because the participants, while willing, are anxious that their business goes unnoticed.  As a result there are very few obvious ways to directly measure illegal transactions and the ONS have been forced to rely on 10-year-old survey data in order to try and estimate the level of prostitution activity in the UK.

But there is a better way of measuring the number of prostitutes than using survey data.  While many of the activities associated with prostitution are illegal in the UK, paying for sex is actually legal and as a result, prostitution services are widely marketed on the web.  We can use to directly count the number of prostitutes who are marketing on the web and attempt a better estimate of the number of prostitutes in the UK.  

Continue reading How much does prostitution contribute to the UK economy?

Every iPhone release follows the hype cycle

Gartner developed the concept of the technology hype cycle as a tool to help explain the different stages that a new technology goes through as it is released into the market and adopted by users.

Every iPhone release takes its own special path through the hype cycle.

Before a release the rumours start about what features the phone will have.  The hype builds to a frenzy that peaks with a glittering presentation of the new device and pictures of queues, as fanboys line up to be the first to try the new product.

We then enter the #gate stage.  Where some “fatal” technical flaw is discovered by users after release and is widely discussed on social media.  We have had antennagate, mapgate and now: bendgate.

We will get over it though and before you know it we will be asking ourselves “when is the next release due?”

Using data to create a personality test that will tell you how to vote in the Scottish referendum


The Scottish referendum is happening today.

The date was set two years ago but it feels like the British public (English, Scottish, Welsh and Northern Irish alike) only just sat up to take notice in the past 10 days.

Screen Shot 2014-09-17 at 22.40.13

The political debate has all of a sudden erupted into noise and confusion and it made me wonder how people should vote.  

Continue reading Using data to create a personality test that will tell you how to vote in the Scottish referendum

My first impressions of Silicon Valley

I can only do this once: I am going to try and write down my first impressions of San Francisco and Silicon Valley.

I must admit that this is not my first time in the Bay area on business but it is the first time that I am here with a view to establishing a permanent home for our company and that brings with it a different mindset that I hope to be able to share with you.

A little bit of background first.  My name is Andrew Fogg.  I am a co-founder of two technology businesses in London: Kusiri and import•io.  I am spending the summer in San Francisco, leading import•io into Silicon Valley.  My objective is to grow the business here, to help secure further financing for the company and to learn from those that have gone before about how to best build a successful technology company.

Over the coming weeks I will give you an insight into how things are done out here, how it is different to London and what can be learnt from the Valley.  So without further ado, this is my view of Silicon Valley.

Continue reading My first impressions of Silicon Valley