How to scrape Understat for football data in Python with requests and BeautifulSoup

Great work man. Appreciate it.

Thank you for providing this tutorial! If I have a list with the match id's I want to scrape (instead of 1 by 1), what are the necessary modifications to the code? I guess that an additional for loop should be written, but don't know where.

Sorry, I would like to ask, I am a beginner, what exactly the aim of scrapping the understat of football data?

Just coming across and had to click that subscribe button. You're so informative I wish you were my prof 😂 awesome work man!

This is really helpful especially for someone starting with football analysis and getting stuck at the initial step of finding the right data. Is there a way to get pass or any event data in general from understat?

The web scraping demo here is fantastic, very clear and easy to apply to other aspects of the website. Top man!

This is an awesome tutorial! Thanks so much!

THANKYOUUU

Great stuff man, which club do you support? Please don't say arsenal

Hi mate. Is there a way to visualise the data at the end

But i dont think understat has any international or CL data right? Just the leagues ig

This had to be done , thanks for sharing your knowledge.

Am a Real Madrid fan and I subscribed!😁😁😁...thanks for sharing...I will be visiting again!

Hi this is a great video, can please scrape lotto data

Hello brother, thanks for the video. i want a scraping project done. Are you able to help please? we can talk privately.

How can i get data manually from a football match please ?

Hi, thanks for the video. I scrapped the shots data from understat, but I am not sure how to convert the X and Y values into X-coordinate, Y-coordinate values to create a shot map. Can you please give an idea.

Hi, can you help to convert the thrid script in the page called "roostersData? I changed from 1 to 2 in scripts, but even changing variables it doesn't work, seems it's a bit different from the shotsData one...thanks!

can you do this method on the page
b e t 3 6 5 ?
I couldn't with the instruction in this video
Delete the spaces between the words

Thank you so much broooooo 😍

Wowwww

Thank you so much for this video

By converting everything to strings surely that means we cant manipulate the numbers since there arnt any numbers just strings

I had this working a while back, but went to run another game, and I'm getting this error:
NameError Traceback (most recent call last)
<ipython-input-6-dbe8a73dafcb> in <module>()
1 res = requests.get(url)
----> 2 soup = BeautifulSoup(res.content, 'lxml')
3 scripts = soup.find_all('script')

NameError: name 'BeautifulSoup' is not defined
Nothing else changed but the match id. Thank you for your tutorials

could you explain better the coordinate system that these dataframe has? i can't understand where is located the origin (x,y)=(0,0), because these coordinates are always positive (>0). Great video btw GJ

😀

Nice. Where I can learn football analytics?
And is possible to land job in football analytics?

Great video! I'm trying to do this in Java, do you know how to do the encode & decode in Java? I'm talking about this line:

encode('utf8').decode('unicode_escape')

Thank you!

Superb content man! Btw I have good memories of Barcelona, my team (Internacional) defeated them in 2006 with Adriano Gabiru's goal.

Thank you very much, man! It is helpful for my graduation work in university

Do you know how I can scrape multiple matches/pages on that website?

Great Video! Congrats! You could get the entire json converted directly to dataframe by doing:

import ast
pd.read_json(json.dumps(ast.literal_eval(str(data_json['h']))))

Hey, I was wondering: if I want to scrape multiple pages, what kind of timeout should I be using between each request? Thanks for the very helpful video

Awesome video bro...help me write a program to alert me when my variable of choice (team) scores or gets a yellow card or wins a corner kick etc. I need to be able to punch in the id of the team and id of variable I want to keep an eye on, hook it up to the internet and let it scrap while Iwait for the program to alert me if id (goal, corner, yellow card, penalty, odd) is True...

U get the idea....

Can I ask what the x and y have for meaning in the match?

Hey man, excelent video!! I started a master in data science and i wanted to practice with something related with football. I will use this for my FPL team

Great tutorial, cheers McKay. Instant new sub!

Guys I get the following error json_data = json_data.encode('uft8').decode('unicode_escape')
LookupError: unknown encoding: uft8. Do you know why I get this error? And how can I solve it

Don't know if this has already been posted, but the nested for loops can be replaced with the following code:

for shot_event in data_home:
x.append(shot_event['X'])
y.append(shot_event['Y'])
xg.append(shot_event['xG'])
team.append(shot_event['h_team'])

And the same for the away team.
Much cleaner imo this way - No nested loops and no multiple ifs.

Github file still exits?

thanks man you saved few hours of my coding

As far as the transformation from json to pd.DataFrame is concerned that one also works :

# Combine 'h' and 'a' dictionaries into a single list
combined_data = data['h'] + data['a']

# Create a DataFrame from the combined data
df = pd.DataFrame(combined_data)

# Display the DataFrame
df

So, it does really create a full data frame from json, having that home/away parameter as a column. Then anyone could try his own cleaning wrangling or usage of understat data himself.

What is that x and y? if those are the x,y coordinates then why does it range from 0-1. Then it will be a square...
please someone help me out with this..

Thank you for taking the time to do this! Been wanting to learn it for a while but lacked the basic skills to start and run run by run. I'd be great if there was a way to just pick a team and start scraping their data from each game for a specific time period... Maybe there's already more work on this as well. Either way I appreciate it!

great video lesson

Thanks McKay, learned a lot from this!

How would you plot this for the shot map

please do a video of scrap data and save to csv file for pizza,radr and other charts.
🙏

Great video! Have you found a way to iterate over the competitions to retrieve all match urls for each competition/season? Or given the structure of Understat we have to manually collect all of them?

Where can I download updated scraped data from the understat website? On github someone shared a package with csv files but last updated 3 years ago. I'm not familiar with Python and can't update the data myself.