If you are researching a youtube video, there are two good python scripts for scraping data on it. There are also several research tools available online for obtaining more information about the video. This post will describe how to use Python and then go into the available research tools.
The Python Scripts and the Youtube API
To use these scripts (or any scripts that scrape youtube data) you first need to get a youtube api key. This article will walk through getting a youtube api key and finding and running the two python scripts on a given video.
Getting a YouTube API key:
First, go to “https://console.developers.google.com/apis/credentials”.
click on agree to terms and services and then click on “agree and continue”
Next click “select a project”
Then click on “new project”
In the next screen click on “create”, and then in the screen click on “select project”, on the left slick on “api & services” and then “credentials”
Then click on “create credentials” and then “API key”
You are not done yet, next click on “library” on the left and in the next screen scroll down to youtube and then click on “youtube data api v3”
In the next screen click “enable”. Now your api has access to youtube data!
To return to your api key, click on “credentials” on the left and you are brought to a page that has your key.
The api documentation is located here – “https://developers.google.com/youtube/v3/docs”
Python Script for Scraping Video Info
There is a python script on Github that will scrape the video’s info and present it in a csv or excel file for you.
To find and run the script, first go here to get set up for the script – “https://github.com/lamthuyvo/social-media-data-scripts/blob/master/README.md”
You will need git, this website will show you how to install it – “https://git-scm.com/book/en/v2/Getting-Started-Installing-Git”
Then go to Terminal or Command Prompt and type the following 3 commands (without the quotes):
“pip install -r requirements.txt”
Then, in Terminal / Command Prompt, navigate to “social-media-data-scripts/01-apis/scripts/”
Find the file named “secrets.py.example” and put your youtube api key in there where it says “youtube api key =”, and then change the file name to “secrets.py”
Now for the video you are researching you need to find it’s video ID. See the screenshot below for an explanation for finding the id.
Next, go to the python script “youtube-get-video-info.py” and input the video id in line 12 where it says “video_ids = “
Run the script with the id of the video (or videos) that you are researching and then an excel file wil appear in the “output” folder in the “01-apis” folder. The excel with have the video’s information for each of the following paramenters:
- commentCount, and
Python Script to Scrape a Video’s Comments
Also, if you want to scrape all of the comments from a video, you can go here – “https://github.com/Jabrils/Download-All-YouTube-Comments-From-Any-Video?files=1”
Then go to the script Dumpallcomments here- “https://github.com/Jabrils/Download-All-YouTube-Comments-From-Any-Video/blob/master/DumpAllComments.py”
To run the script you need to install pytube by typing into terminal “pip install pytube”
copy and paste the script into Sublime Text or whichever python interpreter you are using and put your api key and the video id in lines 7 and 8 as shown below:
Run the script and you will have a tsv (tab separated values) file in the same folder as the script and it will contain all of the video’s comments.
Alternatively, you can click on the “gitpod” button below the script in github and run the script in gitpod. This is pretty simple. You need to first install pytube by typing “pip install pytube” in terminal (at the bottom of the screen in the picture below). Then click on the file “dumpallcomments.py” on the left and then Input the api key and video id into the script.
save the file and then click on the little green arrow on the top right
And a file will appear above “dumpallcomments.py” with your comments. The file contents will look like this screenshot below:
That’s it! You are done!
The Research Tools
This next part will address some of the tools that are available for researching videos online.
What to Look For
If you are researching videos online there are 3 basic research goals to look for.
1 – Search for videos. There are two subreasons to search for videos. 1a – You are looking for information and it might be available on a video, but you don’t know what that video is. In this case you are looking for videos based on topic. 1b -You have already found a video you are very interested in and you want to analyze it but before you can do so, you need to find the original video. Copies of the video may be altered or edited, plus you might be interested in researching the source of the video. To find the original video you must search for other versions of the video online and find the one that was posted online first.
2 – Try to find out who posted the video. This is difficult and basically involves searching if the youtube account username is also used in social media accounts. Also, you can look up the first commenter, or maybe the first few commenters, that posted on the video because they might know the person that posted the video. You can search for social media accounts with their usernames and look into their common friends for potential candidates. Searching for the original video may reveal that it was originally posted on a social media account, which would obviously make the job easier.
3 – Analysis of the video content. This skill is largely outside the bounds of this blog but we identify tools and guides to for this kind of operation. If you are searching videos to see if they have information on a specific topic (like a person, company, legislation) you can now do a word search in youtube to see if anyone says something like mentioning a person, company, or legislation. This feature can help save a lot of time for a researcher going through different videos..
Search for Videos
Searching for videos can be difficult. If you are looking for videos on a certain topic you can try using Petey Vid, a text-based search engine that searches exclusively for videos. Keep in mind that text-based searches, whether they are PeteyVid or Google, can only search for text that is affiliated with a video. So if the video is on a blank webpage and has no affiliated words, the search engine can’t find it.
According to Bellingcat’s Aric Toler, there is currently no way to run an Internet search on a video (this refers to the idea of actually uploading the a video to a search engine that would look for other videos based on it, like a reverse image search). So the next best thing is to get thumbnail photos of the video and run reverse image searches on those photos. The idea is the you are hopefully searching for the photo that appears on a video when it has not yet been started. If this sounds confusing we will walk through an example. At the time of this post’s writing, the youtube homepage looked like this screenshot below:
So if you wanted to search for the video on the top right, you could use the snipping tool to capture the photo that is currently on the video while it has not yet been played. You would do a reverse image search on this photo:
Amnesty International also has a tool for the public called the Youtube DataViewer that extracts data from any youtube video and creates four thumbnail images from the video that you can use for reverse image searches. For example, if we paste the youtube video’s url into the tool and run it, we get the following results.
The results above show the name and description of the video, the video ID, and the specific date and time when it was uploaded.
If we scroll down, we see that the tool created four thumbnail images from the video that can be used for a reverse image search, and there is even a link next to each photo that will do the reverse image search.
to do a reverse image search. There is also a link next to each photo that can be used to do
Try to Find Out Who Posted the Video
This step requires a bit of time but is relatively simple. Google the username
For example, look at this video here. When we click on the user’s ID in the bottom left corner we are brought to a channel homepage.
That brings us to this page below, which has a strange url that does not identify the user well.
Strangely, if you click on “Home” you will get the same webpage the url changes to show the user’s username. See below, the channel’s homepage is the same but the url has chanced to show that the username is “oregonzoo”.
A quick google search of the username reveals the following Twitter account, which of course provides further information on the user.
When we scroll down to the first two commenters we can apply the same method to try to find their social media.
If we find two twitter accounts associated with those commenters we can try to find common friends by using Tweetbeaver.com, see below.
Using this tool we can look for common friends that might be the youtube poster.
Youtube’s Computer-Generated Transcripts
Youtube has a new feature that makes researching videos easier by generating transcripts for each video. To access the feature, click on the three dots below the video on the right side and then click “Open transcript”.
The computer-generated transcript of the video appears next to the video. The transcript is word-searchable so you can save time by searching for a specific name or company is mentioned in the video. in the screenshot below, I searched for the name Tina Larsen and it popped up in the transcript. Note that the transcript also shows the time when different things were said. If you find something interesting in the transcript you can click on the words and youtube will automatically bring the video to that time.
There is a very good guide for analyzing content in a video to do things like find the location or verify the video in the Verification Handbook.