sophocles

Sophocles is a fun way to give your computer some creative writing lessons with AI. Current functionality enables it to write a (mostly nonsensical) story based on Sherlock Holmes, or write a short article on a topic of your choice based on two Wikipedia articles. It's built on @karpathy's vanilla RNN, min-char-rnn as well as on on @weixsong's documentation.

min-char-rnn

Minimal character-level language model with a Vanilla Recurrent Neural Network, in Python/numpy

Reference page The Unreasonable Effectiveness of Recurrent Neural Networks

RNN/LSTM

Actually this model use the simple RNN, not using LSTM. This model use the characters as input, then we use a one-hot vector as input X, the dimension of X is the size of characters in input file.

Let assume that the char vocab size is V, and hidden size is H, then output layer size is V.

This simple RNN model contain 3 matrices:

• Whh: H * H, hidden layer to hidden layer
• Wxh: V * H, input layer to hidden layer
• Why: V * H, hidden layer to output layer

In the output layer, softmax is used to compute the character probability distribution, then we could sample the next character according previous input.

RNN Equation

update hidden state

h^t = tanh(Whh * h^t-1 + Wxh*X)

compute output vector

y = Why * h

Training data

This model use the characters in input.txt file, use current character and next character as a training data pair. Each character is represented by one hot vector, target value means which character we expected given current character.

For example: in input file, we read in "hello", then 'h' and 'e' will be used as a training pair, and will be encoded into vectors. Let's assume that we only have 4 characters in our vocab, ('h','e','l','o'), then 'h' will be encoded to [1,0,0,0]^T, 'e' will be encoded to [0,1,0,0]^T

Input Layer

Input layer size is V, then input value is a V * 1 one hot vector.

Hidden Layer

Hidden layer size is H, we also need to record the hidden state(value of hidden layer).

Output Layer

Output layer size is V, we get a character probability distribution in output layer, then we could sample a character in this probability distribution given a sequence of input.

Cost Function

This RNN model use cross entropy as cost function (error), cross entropy for one training data: $$H(t, y) = -\sum_{i=1}^{V}t_{i}logy_{i}$$

Because in t, most of the value is 0, so, we could rewrite the above equation as: $$H(t, y) = -t_{i}logy_{i}$$

Here i is the indice of 1 in vector t.

Train RNN Model

• Install numpy (basis for the CNN)

pip install numpy

• Install Wikipedia (package for parsing Wikipedia articles)

pip install wikipedia

• Install argparse if you don't already have it

pip install argparse

• Run this code, and follow the instructions onscreen

python min-char-rnn.py

• If you want iterations to be printed to the terminal, use

python min-char-rnn.py --iter

Inputs

• If you type "novel" as an input to Sophocles, the script will use the Sherlock Holmes novel A Study in Scarlet to train its neural network, from the file stud.txt.

• If you type "Wikipedia" as an input, you'll be asked to provide the names of three Wikipedia articles (spaces are allowed). Sophocles will scrape the intro section of each article and use these intros to train its neural network.

• Advanced users only: If you want to use more of each Wikipedia article, or perhaps a different section of it, start by looking at how Wikipedia can be accessed through json.

Output of this model

The output of this model is a couple of text files, each containing 5000 characters of writing generated by the computer. The first is generated after 100k iterations and is called Sophocles100k.txt; the second is generated after 400k iterations and is called Sophocles400k.txt. Remember that if you want to keep these outputs for later, rename the files and/or move them out of this script's directory - otherwise they'll be overwritten!

If you wish to change the length of the output text, change the last number on this line in min-char-rnn.py:

      sample_ix = sample(hprev, inputs[0], 5000)