Python Tutorial-Mining imgur images

Curiosity Bits - curiositybits.com

• This tutorial is created for social scientists interested in grabbing data from the image-hosting site, Imgur (imgur.com).

• Find out more about Python for mining the social web, please visit Curiosity Bits (curiositybits.com).

• Social-Metrics.org also hosts a series of Python tutorials on aggregating and analyzing Twitter/Facebook data. More at social-metrics.org

CURIOSITY BITS©

Get Imgur Data through Python

This tutorial shows you how to download images and the images’ meta-data (i.e., image title, description, source, upload time, etc.).

CURIOSITY BITS©

Imgur.com is a popular site for sharing selfies and images intended for persuasion.

CURIOSITY BITS©

Images will be saved in a designated folder.

The metadata will be saved in a SQLite database

CURIOSITY BITS©

Imgur images are available through keyword search, reddit timelines and public albums.

Here are the examples:• http://imgur.com/search?q=hillary images from the search using the

keyword “hillary”• http://imgur.com/r/transtimelines images from a public reddit timeline

called transtimelines• http://imgur.com/gallery/ROYAZ images from a public album called

Three sources of Imgur images

Final note before we start: for simplicity, I am laying out only the most essential steps. Previous tutorials provide details about how to set up a Python programing environment, please visit curiositybits.com and click the PYTHON tab.

CURIOSITY BITS©

1. Install Anaconda Python (with Spyder and Ipython Notebook)2. Install SQLite Browser3. Install four essential Python packages (imgurpython, sqlalchemy, urllib, sqlite3)4. Register a Imgur client to get client ID and client secret 5. Create a SQLite database for images from keyword search6. Download images from the keyword search 7. Create a SQLite database for images from a reddit timeline8. Download images from a reddit timeline9. Create a SQLite database for images from a public album10. Download images from a public album

CURIOSITY BITS©

Install Anaconda Python

• store.continuum.io/cshop/anaconda/

You can run Python codes in Spyder, which is a component in Anaconda Python

Or use IPython Notebook, a web-based interactive Python environment

CURIOSITY BITS©

This is what Spyder looks like. The left side of the window displays codes. The one at the bottom right shows you the results.

IPython Notebook runs your codes on the web and displays results in your browser.

Install Anaconda Python

CURIOSITY BITS©

http://sqlitebrowser.org

Install SQLite Browser

CURIOSITY BITS©

• In the Command Prompt, use the command line pip install, followed by the package name, to install necessary packages.

Required packages• Imgurpython• Sqlalchemy• Urllib• sqlite3

Install four essential Python packages

CURIOSITY BITS©

type the following command lines in the command prompt:

pip install imgurpythonpip install sqlalchemypip install urllibpip install sqlite3

Install four essential Python packages

Imgur Fetcher V1 is a collection of codes used in this tutorial. You can download them at https://github.com/cosmopolitanvan/imgur_curiositybits_v1

On the site, you can also find three examples of SQLite database. The database files end with the extention .sqlite

CURIOSITY BITS©

The Python codes used in the tutorial

CURIOSITY BITS©

Sign in your Imgur account, go to https://api.imgur.com/oauth2/addclient and get your “security clearance”.

Register a Imgur client

CURIOSITY BITS©

You need Client ID and Client secret to grab data from Imgur API

Register a Imgur client

CURIOSITY BITS©

Let’s get images with the keyword “hilliary”

http://imgur.com/search?q=hillary

We will download all available images, but at first, let’s get their metadata. By metadata, I mean, attributes related to each image. Examples of the metadata include image title, image description, image upload date, image link (which is what we will use to download images).

Create a SQLite database for images from keyword search

You can right-click the link and download the file with the extension .py. Open the file in Spyder.

Or, copy the entire block of codes into Ipython Notebook.

CURIOSITY BITS©

Use the code named Imgur search v1.py

CURIOSITY BITS©

You don’t have to change anything in this block of codes. It is used for importing necessary Python packages. Think of packages as apps running on iOS.

Open Imgur search v1.py in Spyder

CURIOSITY BITS©

This is where you enter the keyword(s) you want to apply to the search. You can have multiple keywords, wrapped in parenthesis, and separated by a comma.

CURIOSITY BITS©

This is where you enter the client ID and client secret generated in the previous step.

CURIOSITY BITS©

No tweaking is needed for this block. But it gives you a sense of what data are to be collected.

CURIOSITY BITS©

Go to line 136, this is where you specify the name of the SQLite database to be saved. If no absolute file path is given, the database will be saved in the same folder with your Python script (Imgur search v1.py).

CURIOSITY BITS©

Imgur images are indexed into multiple pages. Here, 5 means that we are to get five pages of images from the keyword search. You can put 3 or 1 or 2, just play around to see what number gives you the most adequate, while at the same time manageable amount of data.

CURIOSITY BITS©

Now, the tweaking is done! From the menu, chose Run-Configure and hit RUN

CURIOSITY BITS©

This is the SQLite database that contains all the metadata.

CURIOSITY BITS©

Notice that the last column is called multiple_images.

Most image links will end with an extension name of a picture (.jpg, .png, .,bmp, .gif). If that is the case, the value on the column will be NO.

But, some links contain multiple images, and you will see “this link contains multiple images…” warning in the column.

CURIOSITY BITS©

Now, we have a saved SQLite database with image links. The next step is to use Python to download images from those image links

Use the .py file named Imgur search_downloader v1.py

Download images from the keyword search

CURIOSITY BITS©

Go to line 72, and make sure the filename there matches the SQLite database we have just created.

CURIOSITY BITS©

Please create a new folder and enter the folder path here.

Image filename is comprised of the image’s unique identifier id from Imgur API.

Again, this script downloads images from links with the extension of .jpg, .png, .gif, .bmp

If an image link ends with anything but an image extension, the script will prompt a reminder message.

Now, execute the code and you will find the downloaded images in the folder you have just created.

Getting image from a reddit timeline is very similar to getting images from keyword search. This time, we will try executing the Python code in Ipython Notebook.

Use the script called Imgur reddit timeline v1.py

Create a SQLite database for images from a reddit timeline

Create a new notebook in IPython Notebook and copy and paste the code from Imgur reddit timeline v1.py

Create a SQLite database for images from a reddit timeline

Enter the name of a reddit timeline. In the URL term, a reddit timeline displayed ashttp://imgur.com/r/buffalo

Enter your client ID and client secret.

Enter or change the filename of the SQLite database to be saved

The number of pages to be grabbed…

Click Run Cell

The output will be saved in your IPython Notebooks folder. By default, it is …\Documents\IPython Notebooks

Download images from a reddit timeline

Use the .py file named Imgur reddit_downloader v1.py

Load the code in IPython Notebooks, Like what you do to download images from keyword search, make sure the file path in the code matches the database you have just created.

Specify the folder path, and run the code!

Create a SQLite database for images from a public albumExactly as what we do in previous steps. There are only a few places in the script that need tweaking. You need to enter client ID, client secret, the name of the album, the file path of the database, the number of pages to be grabbed. Then you are all good to go!

To create a SQLite database for images from a public album, use the script named Imgur album v1.py

Download images from a public album

To download images from a public album, use the script named Imgur alubm_downloader v1.py

Having questions?

Contact @cosmopolitanvan on Twitter

Python Tutorial-Mining imgur images

Education

The Python Language Reference - New Mexico Institute of Mining and

Assembly Data Mining Platform based on Python

Herramienta Imgur

Python 2.5 quick reference - New Mexico Institute of Mining and

Mining Smartphone Data (with Python)

Camp Imgur - The Internet is a Conspiracy Theory

Imgur - Μπισμπίκη

10 Great Fantasy Books - Imgur

Creative Data Mining 9 | 10 | 2017 · Creative Data Mining ... Introduction to Python & Programming I 9 ... Richard, Will. Building Machine Learning Systems with Python, Packt Publishing

Mining python-software-pyconuk13

Data Mining - PythonDr. Jean-Michel RICHER Data Mining - Python 29 / 37 Run a python program Execution You have two possibilities under Linux use python program.py put as ﬁrst line

Data Mining with Python (Working draft)

Learning Data Mining with Python - Sample Chapter

Big Data Mining in Twitter using Python Clients

Reddit & Imgur (Pop Culture to Watch)

Python Programming in Context Chapter 7. Objectives To use Python lists as a means of storing data To implement a nontrivial data mining application To

Curiosity Bits Tutorial: Mining Twitter User Profile on Python V2

News Aggregation Social Bot using Data Mining - IJCTA · News Aggregation Social Bot using Data Mining . ... by using Stemming Text Mining Python libraries, ... C. Web Scrapping for

CREATIVE DATA MINING - iA · CREATIVE DATA MINING. Introduction to Python and Programming II. 16.10.2017. Continuous output. Supervised learning . Unsupervised learning. Discrete

Maine Mineral Collecting 2018 - Album on Imgur · Maine Mineral Collecting 2018 - Album on Imgur 1 of 18 7/23/2018, 7:28 AM