Creating a Deep Learning Model to Track Trump on TikTok

In today’s article, we’ll walk through how to create a deep learning model to track when Trump appears on the app TikTok. The purpose of this tutorial is to help journalists around the world better monitor the video-sharing app so that they can track trends and break news faster.

This tutorial is largely based on Pulkit Sharma’s tutorial in Analytics Vidhya:

Deep Learning Tutorial to Calculate the Screen Time of Actors in any Video (with Python codes)

All the code for this project can be found at this Github:

jayge2017/DeepLearningTikTok

The Importance of TikTok to Journalists

TikTok Teens and K-Pop Stans Say They Sank Trump Rally

TikTok has become a rising app and an important information source for journalists. Protests are organized on the platform — in June teens and K-Pop stans trended as they falsely registered for a Trump rally to artificially sell out the event. Live protest footage has also been recorded and shared on the app. In addition to political activity, the app has also been used by QAnon to spread conspiracy theories. When a conspiracy theory that Wayfair was selling children started to trend, Wayfair stock prices started to drop.

However despite the importance of the app, it’s notoriously hard to monitor. Unlike Twitter and Facebook, it’s primary media are videos. Though there are hashtags associated with each video, oftentimes they are only loosely related to the topic of the video. In fact, many people add trending hashtags regardless of whether or not they relate to their video in the hopes of making it someone’s For You Page, essentially a user’s home feed of TikTok.

This tutorial shows you how to create a deep learning model to recognize a specific face on the app. The face I have chosen in President Trump, however any face can be used. In fact, any image can be used. For example if you wanted to train the model to recognize confederate flags on the app, you can just swap the training data for images of confederate flags). I will then show you how this model can then be used by a separate python script that will call a TikTok scraping api, and which can be run in the background of any computer, monitoring for potential appearances of Trump on the app. This will allow journalists to be alerted every time the president is trending on the app.

Setup

This project has two steps:

1. Creating the deep learning model
2. Connecting the deep learning model to TikTok api

For Step 1, we will train and create the model from a python notebook on Google Colab. Google Colab is free and we don’t require a particularly large amount of cpu to train our model, so no need to sign up for the Colab Pro account.

Download the training images and mapping from this repository:

jayge2017/DeepLearningTikTok

And upload them all into a folder called Data_deeplearning into your google drive. These training images are captures taken from TikTok videos that I found by searching for videos tagged with Trump. Notice some of them include trump and some of them do not. I’ve gone through the trouble of labeling them so you don’t have to! (Trust me this was by far the most boring and tedious part of this entire project.)

Then, create a Colab notebook in your google drive and mount your notebook onto your drive, then change directory into your Data_deeplearning folder. This will allow your notebook to access all the images in that folder.

from google.colab import drive
import os
drive.mount('/content/drive')
path = "/content/drive/MyDrive/Data_deeplearning"
os.chdir( path )

Preparing the Images

We now need to convert the images into numpy arrays so that our deep learning model can read them later on. To do this, we will read in the file “mapping_trump.csv” in the DeepLearningTikTok directory using pandas, then using that dataframe we will iterate over each image and convert it from an image to a numpy array. We will do the same thing for the labels. Note that from the label you can see 1 means Trump is in the picture and 0 means there is no Trump in that picture.

X = [ ]     # creating an empty array
for img_name in data.Image_ID:
    img = plt.imread('' + img_name)
    X.append(img)  # storing each image in array X
X = np.array(X)    # converting list to array
y = data.Class #set the labels
dummy_y = np_utils.to_categorical(y)    # one hot encoding Classes, we need this to match output shape of model
image = []
for i in range(0,X.shape[0]):
    a = resize(
        X[i], preserve_range=True,output_shape(224,224)
        ).astype(int)
    image.append(a)
X = np.array(image)

Data Augmentation

Because the data was hand-labeled, it’s a relatively small dataset in the deep learning realm. To synthetically increase the size of the dataset we use data augmentation, a technique in deep learning that creates slightly modified copies of the images in order to increase your data set size. Using ImageDataGenerator from the keras library, we can rescale, add to the brightness range, shear some of the photo, zoom in, and horizontally flip the photos.

from tensorflow.keras.preprocessing.image import ImageDataGenerator
import numpy as np
datagen = ImageDataGenerator(
          rescale=1./255,
          brightness_range=[0.2,1.0],
          shear_range=0.2,
          zoom_range=0.2,
          horizontal_flip=True
          )
datagen.fit(X)
X_aug = []
y_aug = []
i = 0
for batch in datagen.flow(X,dummy_y, batch_size = 1):
X_aug.append(batch[0][0])
    y_aug.append(batch[1][0])
    i+=1
    if i>2524:
        break
X = np.append(X,np.array(X_aug),axis=0)
dummy_y = np.append(dummy_y,np.array(y_aug),axis=0)

Preprocessing using the VGG16 model

We will first run the images through the VGG16 base model. VGG16 is a pre-trained model that takes in (224,224) RGB images and converts them into features. It comes out-of-the-box from the keras library and has been trained on millions of images from ImageNet. To read more about VGG16 model check out this site: https://neurohive.io/en/popular-networks/vgg16/

We only run the X data through the VGG16 model to convert it to features because the y data is simply the labels, 0 or 1.

from keras.applications.vgg16 import preprocess_input
from keras.models import Sequential
from keras.applications.vgg16 import VGG16
from keras.layers import Dense, InputLayer, Dropout, Flatten
from sklearn.model_selection import train_test_split
X = preprocess_input(X)

X_train, X_valid, y_train, y_valid = train_test_split(X, dummy_y, test_size=0.3, random_state=42)    # preparing the validation set
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3)) 
X_train = base_model.predict(X_train)
X_valid = base_model.predict(X_valid)
X_train.shape, X_valid.shape

Building Custom Deep Learning Model

In this section we will build a custom deep learning model — this will be the model we use with the TikTok api to predict whether or not a trump appears in a video. In this part, we actually have a lot of freedom to change the structure, add layers, subtract neurons, etc. The important thing is that the input layer of your model matches the dimensions of your X data (which you can see by printing X.shape), that you add a flatten layer before your output layer (so that your multidimensional images can match the dimensions of your label), and that your final layer has the same number of neurons as the number of possible labels in your dataset (in our case it is 0 or 1 so the number of neurons is 2).

#Building the model
model = Sequential()
model.add(InputLayer((7,7,512,))) 
model.add(Flatten())
model.add(Dense(units=1024, activation='sigmoid')) 
model.add(Dropout(0.5))      # adding dropout prevents overfitting
model.add(Dense(units=512, activation='sigmoid'))    # hidden layer
model.add(Dropout(0.5))      # adding dropout
model.add(Dense(units=512, activation='sigmoid'))    # hidden layer
model.add(Dropout(0.5))      # adding dropout
model.add(Dense(units=256, activation='sigmoid'))    # hidden layer
model.add(Dropout(0.5))      # adding dropout
model.add(Dense(2, activation='softmax'))

Now we train the model using our training images from before:

model.fit(X_train, y_train, steps_per_epoch=len(X_train) / 32, epochs = 100, validation_data = (X_valid, y_valid))

Finally save the model so that we can download to a desktop and connect it with the TikTok api:

model.save(‘trump_model_final.h5’)

Now you have a model that can recognize when Trump is in tiktok videos! The next step is to connect it to a tiktok api.

Now for this part you will need to clone the github repository:

jayge2017/DeepLearningTikTok

After cloning this repo into your local, move your model (trump_model_final.h5) that you downloaded from the previous step into this repo. Then, you can test it out by calling it with the name of a video after. For example, if you want to test with the biden.mp4 that is placed in the repo as an example, simply call:

python3 run_model.py Biden.mp4

You should see this output:

Prediction: [0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 

Each number stands for a different frame taken of the video. The 0’s mean the model did not detect any images of Trump, and the 1’s mean that there was a Trump detection in that frame. Notice that in this video, there is only images of Biden, however the model still picked up some false positives. This is a good reminder that this is a very rudimentary model, and to improve accuracy you need to expand the training dataset. To account for this, I only consider when a model has detected Trump in a third of all the frames to be a “positive trump detection”. If you want to try the model with the example Trump video in the repo, run:

python3 run_model.py Trump.mp4

You should now see the output to be:

Prediction: [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
====================Trump is in this video==================

Connecting with TikTok api

Now that you have the model which recognizes Trump, the python script which feeds an mp4 file video into the model and outputs whether it detects Trump, the only thing missing is the TikTok api. For this part I used drawrowfly’s open source tiktok scraper.

drawrowfly/tiktok-scraper

Thank you to drawrowfly for their open source work!

I’ve written a really rudimentary bash shell script that just calls the TikTok api scraper to get the most recent trending video, and then feeds that filepath to the python script, which will evaluate if more than 1/3 of the frames has Trump in it. To convert this into a 24/7 running file, you can just wrap it all in a while loop or write a cron job to regularly schedule it to run.

Appendix

The dataset that is included in the git repo under Data_deeplearning is a combination of tiktok videos I took captures of that included Trump, tiktok videos I took captures of that included other notable politicians such as Biden, tiktok videos that were trending, and google images of Trump. I found that using only one of the listed sources made the model less effective.

To learn more about deep learning, you can read more here:

What is Deep Learning? - Machine Learning Mastery