Category: Machine learning

Recurrent neural networks (RNNs) are a type of neural network architecture that are well-suited for processing sequential data. One popular variant of RNNs is the long short-term memory (LSTM) unit, which is capable of learning and maintaining long-term dependencies in the data.

Stacked LSTM is a type of RNN architecture that involves stacking multiple LSTM layers on top of each other. This allows the network to learn more complex patterns in the data and make more accurate predictions. Stacked LSTMs are commonly used in tasks such as language modeling, machine translation, and speech recognition.

In this article, we will introduce the concept of stacked LSTM and discuss how it can be implemented using the Keras library in Python. We will also cover some best practices for using stacked LSTM and some potential challenges to be aware of. Finally, we will provide some examples of applications where stacked LSTM has been used successfully.

import numpy as np
import tensorflow as tf

cells = []

LSTM_CELL_SIZE_1 = 4 #4 hidden nodes
cell1 = tf.keras.layers.LSTMCell(LSTM_CELL_SIZE_1)
cells.append(cell1)

LSTM_CELL_SIZE_2 = 5 #5 hidden nodes
cell2 = tf.keras.layers.LSTMCell(LSTM_CELL_SIZE_2)
cells.append(cell2)

stacked_lstm =  tf.keras.layers.StackedRNNCells(cells)

lstm_layer = tf.keras.layers.RNN(stacked_lstm, return_sequences=True, return_state=True)

#Batch size x time steps x features.
sample_input = [[[1,2,3,4,3,2], [1,2,1,1,1,2],[1,2,2,2,2,2]],[[1,2,3,4,3,2],[3,2,2,1,1,2],[0,0,0,0,3,2]]]
sample_input

batch_size = 2
time_steps = 3
features = 6
new_shape = (batch_size, time_steps, features)

x = tf.constant(np.reshape(sample_input, new_shape), dtype = tf.float32)

output, final_memory_state, final_carry_state  = lstm_layer(x)

print('Output : ', tf.shape(output))

print('Memory : ',tf.shape(final_memory_state))

print('Carry state : ',tf.shape(final_carry_state))

This code defines a stacked LSTM with two layers, each with a different number of hidden nodes (4 and 5). The tf.keras.layers.LSTMCell class is used to define the individual LSTM cells, and these cells are then stacked using the tf.keras.layers.StackedRNNCells class.

The tf.keras.layers.RNN class is then used to define the RNN layer that wraps the stacked LSTM cells. The return_sequences argument specifies whether the output of the RNN layer should be returned as a sequence (i.e., a 3D tensor with shape (batch_size, timesteps, units)) or as a single vector (i.e., a 2D tensor with shape (batch_size, units)). In this case, the output is set to be returned as a sequence. The return_state argument specifies whether the final memory and carry states of the stacked LSTM should be returned as well.

The input data, x, is then passed through the stacked LSTM layer using the call method. The output, final memory state, and final carry state are returned as a tuple. The shapes of the output, final memory state, and final carry state are printed using the tf.shape function. The output should have shape (batch_size, timesteps, units), where units is the number of hidden nodes in the final LSTM layer (in this case, 5). The final memory state and final carry state should both have shape (batch_size, units).

cells = [] 
LSTM_CELL_SIZE_1 = 4 #4 hidden nodes cell1 = tf.keras.layers.LSTMCell(LSTM_CELL_SIZE_1) cells.append(cell1) LSTM_CELL_SIZE_2 = 5 #5 hidden nodes cell2 = tf.keras.layers.LSTMCell(LSTM_CELL_SIZE_2) cells.append(cell2)

In the above code, two LSTM cells are created and added to a list called cells. The first cell has 4 hidden nodes, and the second cell has 5 hidden nodes.

An LSTM cell is a type of memory cell that is commonly used in RNNs to process sequential data. It has three gates (input, output, and forget) that control the flow of information into and out of the cell, allowing it to maintain long-term dependencies in the data.

The tf.keras.layers.LSTMCell class is used to define the LSTM cells. It takes the number of hidden nodes (also called units) as an argument. In this case, the first cell has 4 units and the second cell has 5 units.

After the LSTM cells are created, they are added to the cells list. This list will be used later to define a stacked LSTM, which is an RNN architecture that consists of multiple LSTM layers stacked on top of each other.

stacked_lstm =  tf.keras.layers.StackedRNNCells(cells)

lstm_layer = tf.keras.layers.RNN(stacked_lstm, return_sequences=True, return_state=True)

In the above code, a stacked LSTM is defined using the tf.keras.layers.StackedRNNCells class and the list of LSTM cells (cells) that was created in the previous code snippet. The tf.keras.layers.StackedRNNCells class takes a list of RNN cells as an argument and stacks them on top of each other to form a multi-layer RNN.

Next, an RNN layer is defined using the tf.keras.layers.RNN class and the stacked LSTM cells (stacked_lstm). The return_sequences argument specifies whether the output of the RNN layer should be returned as a sequence (i.e., a 3D tensor with shape (batch_size, timesteps, units)) or as a single vector (i.e., a 2D tensor with shape (batch_size, units)). In this case, the output is set to be returned as a sequence. The return_state argument specifies whether the final memory and carry states of the stacked LSTM should be returned as well.

The resulting RNN layer can then be used to process sequential data by calling the call method and passing in the input data. The output, final memory state, and final carry state will be returned as a tuple.

#Batch size x time steps x features.
sample_input = [[[1,2,3,4,3,2], [1,2,1,1,1,2],[1,2,2,2,2,2]],[[1,2,3,4,3,2],[3,2,2,1,1,2],[0,0,0,0,3,2]]]
sample_input

batch_size = 2
time_steps = 3
features = 6
new_shape = (batch_size, time_steps, features)

In the above code, a sample input for the stacked LSTM is defined as a list of lists of lists. The outermost list has a length of 2, which corresponds to the batch size. The second level of lists has a length of 3, which corresponds to the number of time steps. The innermost lists have a length of 6, which corresponds to the number of features at each time step.

The shape of the input data is then defined as a tuple with three elements: batch_size, time_steps, and features. This shape will be used later to reshape the sample input into the correct format for processing by the stacked LSTM.

It’s worth noting that the sample input in this code snippet is just an example and may not represent a realistic input for a real-world task. The actual input shape and content will depend on the specific task you are trying to solve.

In machine learning and deep learning, the model does not actually “learn” all of the samples in a batch at the same time. Instead, the model processes the samples in the batch one at a time, using the output of the previous sample to inform the processing of the next sample.

For example, if the batch size is set to 32 and the model is being trained to classify images, the model will process the first image in the batch, make a prediction based on that image, and use the prediction to compute the loss and gradients. It will then move on to the second image in the batch and repeat the process. This process will continue until all 32 images in the batch have been processed.

The model then uses the gradients computed from the batch to update its parameters, and this process is repeated for each batch of data during training. The goal is to find the set of model parameters that minimize the loss function and enable the model to make accurate predictions on unseen data.

So while the model does not learn all of the samples in a batch at the same time, it does use the information from all of the samples in the batch to inform the learning process. The batch size simply determines how many samples are processed together at a time during training, which can impact the performance and efficiency of the model.

In machine learning and deep learning, an iteration refers to a single pass through the training data during the training process. During each iteration, the model processes a batch of data, computes the gradients based on the loss function, and updates the model parameters. This process is repeated for each batch of data in the training set until all of the data has been processed.

The number of iterations that are performed during training is often referred to as the number of epochs. For example, if the batch size is set to 32 and the training set contains 10,000 samples, then there will be 10,000/32 = 312.5 iterations per epoch. If the number of epochs is set to 10, then the model will perform a total of 312.5 * 10 = 3125 iterations during training.

The goal of the training process is to find the set of model parameters that minimize the loss function and enable the model to make accurate predictions on unseen data. The number of iterations and epochs can impact the performance of the model, as well as the training time. In general, more iterations can lead to a better fit to the training data, but it can also increase the risk of overfitting if the model is not properly regularized.

x = tf.constant(np.reshape(sample_input, new_shape), dtype = tf.float32)

output, final_memory_state, final_carry_state  = lstm_layer(x)

print('Output : ', tf.shape(output))

print('Memory : ',tf.shape(final_memory_state))

print('Carry state : ',tf.shape(final_carry_state))

In the above code, the sample input (sample_input) is reshaped using the np.reshape function and the new_shape tuple that was defined earlier. The reshaped input is then converted to a tensor using the tf.constant function.

The reshaped input tensor (x) is then passed through the stacked LSTM layer using the call method. The output, final memory state, and final carry state are returned as a tuple.

The shapes of the output, final memory state, and final carry state are then printed using the tf.shape function. The output should have shape (batch_size, timesteps, units), where units is the number of hidden nodes in the final LSTM layer (in this case, 5). The final memory state and final carry state should both have shape (batch_size, units).

It’s worth noting that this code is just an example and may not represent a realistic scenario for a real-world task. The actual input shape, output shape, and number of hidden units will depend on the specific task you are trying to solve.

In conclusion, stacked LSTM is a powerful variant of RNN architecture that can be used to process sequential data. By stacking multiple LSTM layers, the network can learn more complex patterns in the data and make more accurate predictions. However, it is important to properly regularize the model to avoid overfitting and to choose the appropriate number of layers and hidden units for the task at hand.

Stacked LSTM has been successfully applied to a range of tasks, including language modeling, machine translation, and speech recognition. By following the steps outlined in this article, you should now have a good understanding of how to implement stacked LSTM using the Keras library in Python. With this knowledge, you can start exploring the capabilities of stacked LSTM and applying it to your own projects.

In the example, we provide the code that defines several functions and variables related to creating a convolutional neural network (CNN) model for image classification. The CNN model has two convolutional layers, each followed by a ReLU activation function and a max pooling layer, and a fully connected layer followed by a softmax activation function. The model is trained using the Adam optimizer with a specified learning rate and is evaluated using cross-entropy loss.

The code loads the MNIST dataset, preprocesses the data by normalizing the feature values, one-hot encodes the labels and converts the data into a format that can be used with TensorFlow’s tf.data module. The tf.data module is used to create a dataset object that can be used to efficiently feed data into a model during training.

Finally, the code defines a training loop that trains the model on the training data and evaluates it on the test data. The training loop includes a summary writer and a TensorBoard callback, which can be used to visualize the training process using TensorBoard.

Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are a type of neural network designed specifically for image classification tasks. They are widely used in a variety of applications including object recognition, face detection, and medical image analysis.

CNNs consist of several layers of interconnected neurons, with each layer responsible for learning a specific set of features from the input data. The first layer of a CNN typically consists of convolutional filters that learn local features from the input image, such as edges and corners. The output of these convolutional filters is then passed through a pooling layer, which downsamples the output and helps to reduce the dimensionality of the data.

The output of the pooling layer is then fed into one or more fully connected (FC) layers, which learn more abstract features from the data and make the final prediction using a softmax activation function. The output of the softmax function is a probability distribution over the possible classes, with the class with the highest probability being the model’s prediction.

During the training process, the weights and biases of the CNN are adjusted using an optimization algorithm, such as Adam or Stochastic Gradient Descent (SGD), in order to minimize a loss function that measures the difference between the predicted and true labels. The trained CNN can then be used to make predictions on new, unseen data.

How to apply CNN to classify MNIST dataset

import tensorflow as tf
from IPython.display import Markdown, display

def printmd(string):
    display(Markdown('# '+string+''))

#Import the MNIST dataset using TensorFlow built-in feature
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()


#The features data are between 0 and 255, and we will normalize this to improve optimization performance.
x_train, x_test = x_train / 255.0, x_test / 255.0


#Let's take a look at the first few label values:
print(y_train[0:5])

#the same digits using one-hot vector representation can be show as:
print("categorical labels")
print(y_train[0:5])

# make labels one hot encoded
y_train = tf.one_hot(y_train, 10)
y_test = tf.one_hot(y_test, 10)

print("one hot encoded labels")
print(y_train[0:5])

print("number of training examples:" , x_train.shape[0])
print("number of test examples:" , x_test.shape[0])

train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(50)
test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(50)

#Converting a 2D Image into a 1D Vector
# showing an example of the Flatten class and operation
from tensorflow.keras.layers import Flatten
flatten = Flatten(dtype='float32')

"original data shape"
print(x_train.shape)

"flattened shape"
print(flatten(x_train).shape)

width = 28 # width of the image in pixels 
height = 28 # height of the image in pixels
flat = width * height # number of pixels in one image 
class_output = 10 # number of possible classifications for the problem

x_image_test = tf.reshape(x_test, [-1,28,28,1]) 
x_image_test = tf.cast(x_image_test, 'float32') 

#creating new dataset with reshaped inputs
train_ds2 = tf.data.Dataset.from_tensor_slices((x_image_train, y_train)).batch(50)
test_ds2 = tf.data.Dataset.from_tensor_slices((x_image_test, y_test)).batch(50)

x_image_train = tf.slice(x_image_train,[0,0,0,0],[10000, 28, 28, 1])
y_train = tf.slice(y_train,[0,0],[10000, 10])

W_conv1 = tf.Variable(tf.random.truncated_normal([5, 5, 1, 32], stddev=0.1, seed=0))
b_conv1 = tf.Variable(tf.constant(0.1, shape=[32])) # need 32 biases for 32 outputs

def convolve1(x):
    return(
        tf.nn.conv2d(x, W_conv1, strides=[1, 1, 1, 1], padding='SAME') + b_conv1)

def h_conv1(x): return(tf.nn.relu(convolve1(x)))

def conv1(x):
    return tf.nn.max_pool(h_conv1(x), ksize=[1, 2, 2, 1], 
                          strides=[1, 2, 2, 1], padding='SAME')

W_conv2 = tf.Variable(tf.random.truncated_normal([5, 5, 32, 64], stddev=0.1, seed=1))
b_conv2 = tf.Variable(tf.constant(0.1, shape=[64])) #need 64 biases for 64 outputs

def convolve2(x): 
    return( 
    tf.nn.conv2d(conv1(x), W_conv2, strides=[1, 1, 1, 1], padding='SAME') + b_conv2)

def h_conv2(x):  return tf.nn.relu(convolve2(x))

def conv2(x):  
    return(
    tf.nn.max_pool(h_conv2(x), ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME'))

def layer2_matrix(x): return tf.reshape(conv2(x), [-1, 7 * 7 * 64])

W_fc1 = tf.Variable(tf.random.truncated_normal([7 * 7 * 64, 1024], stddev=0.1, seed = 2))
b_fc1 = tf.Variable(tf.constant(0.1, shape=[1024])) # need 1024 biases for 1024 outputs

def fcl(x): return tf.matmul(layer2_matrix(x), W_fc1) + b_fc1

def h_fc1(x): return tf.nn.relu(fcl(x))

keep_prob=0.5
def layer_drop(x): return tf.nn.dropout(h_fc1(x), keep_prob)


W_fc2 = tf.Variable(tf.random.truncated_normal([1024, 10], stddev=0.1, seed = 2)) #1024 neurons
b_fc2 = tf.Variable(tf.constant(0.1, shape=[10])) # 10 possibilities for digits [0,1,2,3,4,5,6,7,8,9]


def fc(x): return tf.matmul(layer_drop(x), W_fc2) + b_fc2

def y_CNN(x): return tf.nn.softmax(fc(x))

import numpy as np
layer4_test =[[0.9, 0.1, 0.1],[0.9, 0.1, 0.1]]
y_test=[[1.0, 0.0, 0.0],[1.0, 0.0, 0.0]]
np.mean( -np.sum(y_test * np.log(layer4_test),1))

def cross_entropy(y_label, y_pred):
    return (-tf.reduce_sum(y_label * tf.math.log(y_pred + 1.e-10)))


optimizer = tf.keras.optimizers.Adam(1e-4)

variables = [W_conv1, b_conv1, W_conv2, b_conv2, 
             W_fc1, b_fc1, W_fc2, b_fc2, ]

def train_step(x, y):
    with tf.GradientTape() as tape:
        current_loss = cross_entropy( y, y_CNN( x ))
        grads = tape.gradient( current_loss , variables )
        optimizer.apply_gradients( zip( grads , variables ) )
        return current_loss.numpy()
    
    
correct_prediction = tf.equal(tf.argmax(y_CNN(x_image_train), axis=1), tf.argmax(y_train, axis=1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float32'))

loss_values=[]
accuracies = []
epochs = 1

for i in range(epochs):
    j=0
    # each batch has 50 examples
    for x_train_batch, y_train_batch in train_ds2:
        j+=1
        current_loss = train_step(x_train_batch, y_train_batch)
        if j%50==0: #reporting intermittent batch statistics
            correct_prediction = tf.equal(tf.argmax(y_CNN(x_train_batch), axis=1),
                                  tf.argmax(y_train_batch, axis=1))
            #  accuracy
            accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)).numpy()
            print("epoch ", str(i), "batch", str(j), "loss:", str(current_loss),
                     "accuracy", str(accuracy)) 
            
    current_loss = cross_entropy( y_train, y_CNN( x_image_train )).numpy()
    loss_values.append(current_loss)
    correct_prediction = tf.equal(tf.argmax(y_CNN(x_image_train), axis=1),
                                  tf.argmax(y_train, axis=1))
    #  accuracy
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)).numpy()
    accuracies.append(accuracy)
    print("end of epoch ", str(i), "loss", str(current_loss), "accuracy", str(accuracy) )  
    
   
    
j=0
acccuracies=[]
# evaluate accuracy by batch and average...reporting every 100th batch
for x_train_batch, y_train_batch in train_ds2:
        j+=1
        correct_prediction = tf.equal(tf.argmax(y_CNN(x_train_batch), axis=1),
                                  tf.argmax(y_train_batch, axis=1))
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)).numpy()
        #accuracies.append(accuracy)
        if j%100==0:
            print("batch", str(j), "accuracy", str(accuracy) ) 
import numpy as np
print("accuracy of entire set", str(np.mean(accuracies)))    

The code then defines a function called train() that takes in several arguments:

• num_epochs: the number of epochs to train the model for
• train_ds: the training dataset
• test_ds: the test dataset
• loss_object: an object representing the loss function to use during training
• optimizer: the optimizer to use during training
• train_loss_results: a list to store the training loss results for each epoch
• train_accuracy_results: a list to store the training accuracy results for each epoch
• test_loss_results: a list to store the test loss results for each epoch
• test_accuracy_results: a list to store the test accuracy results for each epoch

The train() function first initializes some variables, such as the epoch number and the training and test datasets. It then enters a loop that iterates over the specified number of epochs. For each epoch, the function iterates over the training dataset and updates the model’s parameters using the optimizer. It also updates the training loss and accuracy results for the epoch.

After the training epoch is complete, the function evaluates the model on the test dataset and updates the test loss and accuracy results. The function then prints the training and test results for the epoch and returns the final training and test results.

The code then calls the train() function to train the model for a specified number of epochs. It also creates a summary writer and a TensorBoard callback, which can be used to visualize the training process using TensorBoard.

Finally, the code plots the training and test loss and accuracy results to allow you to visualize how the model’s performance changed over the course of training.

Explain the code

x_image_train = tf.reshape(x_train, [-1,28,28,1])  
x_image_train = tf.cast(x_image_train, 'float32') 

x_image_test = tf.reshape(x_test, [-1,28,28,1]) 
x_image_test = tf.cast(x_image_test, 'float32') 

In the above code , the x_image_train and x_image_test variables are created by reshaping and casting the x_train and x_test variables, respectively.

The reshaping is done using the tf.reshape() function, which reshapes the input tensor into a new shape. In this case, the input tensor is x_train or x_test, and the new shape is specified as [-1,28,28,1], which means that the tensor will be reshaped into a 4D tensor with shape [batch_size, height, width, num_channels], where batch_size is the size of the batch, height and width are the dimensions of the image, and num_channels is the number of channels (1 in this case, since the images are grayscale). The -1 value in the shape indicates that the size of the batch is determined automatically based on the size of the input tensor.

The casting is done using the tf.cast() function, which casts the input tensor to a new data type. In this case, the input tensor is x_image_train or x_image_test, and the new data type is ‘float32’. This is done to ensure that the tensor has the correct data type for use in the model.

After these operations, the x_image_train and x_image_test variables will contain the training and test data, respectively, in the form of 4D tensors with shape [batch_size, height, width, num_channels] and data type float32. These tensors can be used as input to the CNN model defined in the code.

train_ds2 = tf.data.Dataset.from_tensor_slices((x_image_train, y_train)).batch(50)
test_ds2 = tf.data.Dataset.from_tensor_slices((x_image_test, y_test)).batch(50)

In above code, the train_ds2 and test_ds2 variables are created using the tf.data.Dataset.from_tensor_slices() function, which creates a dataset object from a tensor by slicing it along its first dimension.

The from_tensor_slices() function takes in a tuple of tensors as input, and returns a dataset object that can be used to iterate over the tensors in slices. In this case, the input tensors are x_image_train and y_train for the train_ds2 dataset, and x_image_test and y_test for the test_ds2 dataset.

The batch() function is then used to batch the data from the dataset object. This function groups the data into batches, with each batch having the specified batch size. In this case, the batch size is 50, which means that each batch will contain 50 examples from the dataset.

After these operations, the train_ds2 and test_ds2 variables will contain the training and test data, respectively, in the form of datasets that can be used to efficiently feed the data into the model during training and evaluation.

x_image_train = tf.slice(x_image_train,[0,0,0,0],[10000, 28, 28, 1])
y_train = tf.slice(y_train,[0,0],[10000, 10])

In the above code, the x_image_train and y_train variables are sliced using the tf.slice() function. This function slices a tensor along its dimensions to extract a subset of the tensor.

The tf.slice() function takes in the following arguments:

• input_: The input tensor to slice.
• begin: A list of int or int32 tensors representing the starting indices of the slice for each dimension.
• size: A list of int or int32 tensors representing the size of the slice for each dimension.

In this case, the input tensor is x_image_train for the x_image_train slice, and y_train for the y_train slice. The begin argument is a list of starting indices, with each index representing the starting position for each dimension of the tensor. The size argument is a list of sizes, with each size representing the number of elements to include in the slice for each dimension.

After slicing, the x_image_train and y_train variables will contain the first 10000 examples from the original x_image_train and y_train tensors, respectively. These slices can be used to train the model on a subset of the original data.

W_conv1 = tf.Variable(tf.random.truncated_normal([5, 5, 1, 32], stddev=0.1, seed=0))
b_conv1 = tf.Variable(tf.constant(0.1, shape=[32])) # need 32 biases for 32 outputs

def convolve1(x):
    return(
        tf.nn.conv2d(x, W_conv1, strides=[1, 1, 1, 1], padding='SAME') + b_conv1)

def h_conv1(x): return(tf.nn.relu(convolve1(x)))

def conv1(x):
    return tf.nn.max_pool(h_conv1(x), ksize=[1, 2, 2, 1], 
                          strides=[1, 2, 2, 1], padding='SAME')

W_conv2 = tf.Variable(tf.random.truncated_normal([5, 5, 32, 64], stddev=0.1, seed=1))
b_conv2 = tf.Variable(tf.constant(0.1, shape=[64])) #need 64 biases for 64 outputs

def convolve2(x): 
    return( 
    tf.nn.conv2d(conv1(x), W_conv2, strides=[1, 1, 1, 1], padding='SAME') + b_conv2)

def h_conv2(x):  return tf.nn.relu(convolve2(x))

def conv2(x):  
    return(
    tf.nn.max_pool(h_conv2(x), ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME'))

def layer2_matrix(x): return tf.reshape(conv2(x), [-1, 7 * 7 * 64])

In the above code, a series of functions are defined that perform convolution, activation, and pooling operations on the input tensor. These functions are used to create the first two convolutional layers of the CNN model.

The convolve1() function performs convolution on the input tensor using the W_conv1 convolutional kernel and the b_conv1 bias vector. The strides argument specifies the stride of the kernel as it moves over the input tensor, and the padding argument specifies whether to pad the input tensor with zeros to prevent the kernel from going outside the bounds of the tensor. In this case, the stride is 1 and the padding is ‘SAME’, which means that the kernel will move over the input tensor in steps of 1 pixel and the input tensor will be padded with zeros to ensure that the kernel can be applied to all pixels of the input tensor.

The h_conv1() function applies the ReLU activation function to the output of the convolve1() function. The ReLU activation function is defined as f(x) = max(0, x), which means that it sets all negative values of x to 0 and leaves all positive values unchanged. This function is used to introduce non-linearity into the model.

The conv1() function applies max pooling to the output of the h_conv1() function. Max pooling is a down-sampling operation that reduces the spatial size of the input tensor by taking the maximum value of each subregion of the tensor. The ksize argument specifies the size of the pooling window, and the strides argument specifies the stride of the window as it moves over the input tensor. In this case, the pooling window has a size of 2×2 and a stride of 2, which means that it will take the maximum value of each 2×2 subregion of the input tensor and the convolve2() function is similar to the convolve1() function, but it performs convolution on the output of the conv1() function using the W_conv2 convolutional kernel and the b_conv2 bias vector.

The h_conv2() function is similar to the h_conv1() function, but it applies the ReLU activation function to the output of the convolve2() function.

The conv2() function is similar to the conv1() function, but it applies max pooling to the output of the h_conv2() function.

The layer2_matrix() function reshapes the output of the conv2() function into a 2D matrix with shape [batch_size, 7 * 7 * 64]. This matrix will be used as the input to the fully-connected layers of the CNN model.

These functions will be used to create the first two convolutional layers of the CNN model, which will extract features from the input image and reduce its spatial size. The output of these layers will be fed into the fully-connected layers of the model, which will use the extracted features to make predictions.

W_fc1 = tf.Variable(tf.random.truncated_normal([7 * 7 * 64, 1024], stddev=0.1, seed = 2))
b_fc1 = tf.Variable(tf.constant(0.1, shape=[1024])) # need 1024 biases for 1024 outputs

In the above code, the W_fc1 and b_fc1 variables are defined using the tf.Variable and tf.constant functions, respectively. These variables are used in the first fully-connected (FC) layer of the CNN model.

The W_fc1 variable is a weight matrix that connects the input to the FC layer to the output of the FC layer. It has a shape of [7 * 7 * 64, 1024], which means that it has 7x7x64 input units and 1024 output units. The stddev argument specifies the standard deviation of the truncated normal distribution from which the initial values of the weights are drawn, and the seed argument specifies a seed for the random number generator used to draw the initial values.

The b_fc1 variable is a bias vector with 1024 elements, one for each output unit of the FC layer. The shape argument specifies the shape of the bias vector, and the tf.constant() function is used to create a constant tensor with the specified shape and a constant value of 0.1.

These variables will be used in the first FC layer of the CNN model to transform the input from the second convolutional layer into a higher-level representation that is used to make predictions.

def fcl(x): return tf.matmul(layer2_matrix(x), W_fc1) + b_fc1

def h_fc1(x): return tf.nn.relu(fcl(x))

keep_prob=0.5
def layer_drop(x): return tf.nn.dropout(h_fc1(x), keep_prob)

In the above code, a series of functions are defined that perform operations on the input tensor to create the first fully-connected (FC) layer of the CNN model.

The fcl() function performs the matrix multiplication between the input to the FC layer and the W_fc1 weight matrix, and adds the b_fc1 bias vector. This operation transforms the input from the second convolutional layer into a higher-level representation that is used to make predictions.

The h_fc1() function applies the ReLU activation function to the output of the fcl() function. The ReLU activation function is defined as f(x) = max(0, x), which means that it sets all negative values of x to 0 and leaves all positive values unchanged. This function is used to introduce non-linearity into the model.

The layer_drop() function applies dropout to the output of the h_fc1() function. Dropout is a regularization technique that randomly sets a fraction of the input units to 0 during training to prevent overfitting. The keep_prob variable specifies the probability that an input unit will be kept. In this case, the keep_prob variable is set to 0.5, which means that on average, half of the input units will be kept.

These functions will be used to create the first FC layer of the CNN model, which will transform the input from the second convolutional layer into a higher-level representation that is used to make predictions. The output of this layer will be fed into the next FC layer or the output layer of the model, depending on the architecture of the model.

W_fc2 = tf.Variable(tf.random.truncated_normal([1024, 10], stddev=0.1, seed = 2)) #1024 neurons
b_fc2 = tf.Variable(tf.constant(0.1, shape=[10])) # 10 possibilities for digits [0,1,2,3,4,5,6,7,8,9]

In the above code, the W_fc2 and b_fc2 variables are defined using the tf.Variable and tf.constant functions, respectively. These variables are used in the second fully-connected (FC) layer or the output layer of the CNN model, depending on the architecture of the model.

The W_fc2 variable is a weight matrix that connects the input to the FC layer or output layer to the output of the FC layer or output layer. It has a shape of [1024, 10], which means that it has 1024 input units and 10 output units. The stddev argument specifies the standard deviation of the truncated normal distribution from which the initial values of the weights are drawn, and the seed argument specifies a seed for the random number generator used to draw the initial values.

The b_fc2 variable is a bias vector with 10 elements, one for each output unit of the FC layer or output layer. The shape argument specifies the shape of the bias vector, and the tf.constant() function is used to create a constant tensor with the specified shape and a constant value of 0.1.

These variables will be used in the second FC layer or output layer of the CNN model to transform the input from the first FC layer or the output of the dropout layer into a higher-level representation that is used to make predictions. If the model has a second FC layer, the output of this layer will be fed into the output layer, which will use the extracted features to make predictions. If the model does not have a second FC layer, the output layer will use the output of the dropout layer to make predictions.

def fc(x): return tf.matmul(layer_drop(x), W_fc2) + b_fc2

def y_CNN(x): return tf.nn.softmax(fc(x))

In the above code, the fc() and y_CNN() functions are defined. These functions are used to create the second fully-connected (FC) layer or the output layer of the CNN model, depending on the architecture of the model.

The fc() function performs the matrix multiplication between the input to the FC layer or output layer and the W_fc2 weight matrix, and adds the b_fc2 bias vector. This operation transforms the input from the first FC layer or the output of the dropout layer into a higher-level representation that is used to make predictions.

The y_CNN() function applies the softmax activation function to the output of the fc() function. The softmax activation function is defined as

f(x) = exp(x_i) / sum(exp(x_i))

where x_i is the i-th element of the input tensor x. The softmax function normalizes the output of the FC layer or output layer so that it sums to 1, which makes it suitable for use as a probability distribution over the possible classes.

These functions will be used to create the second FC layer or output layer of the CNN model, which will transform the input from the first FC layer or the output of the dropout layer into a higher-level representation that is used to make predictions. If the model has a second FC layer, the output of this layer will be fed into the output layer, which will use the extracted features to make predictions. If the model does not have a second FC layer, the output layer will use the output of the dropout layer to make predictions.

import numpy as np
layer4_test =[[0.9, 0.1, 0.1],[0.9, 0.1, 0.1]]
y_test=[[1.0, 0.0, 0.0],[1.0, 0.0, 0.0]]
np.mean( -np.sum(y_test * np.log(layer4_test),1))

In the above code, the layer4_test and y_test variables are defined as 2D numpy arrays. The layer4_test array has two rows and three columns, and the y_test array has two rows and three columns.

The np.mean() function is applied to the result of -np.sum(y_test * np.log(layer4_test),1), which calculates the mean cross-entropy loss between the layer4_test array and the y_test array.

The cross-entropy loss is a measure of the difference between the predicted probability distribution and the true probability distribution. In this case, the predicted probability distribution is represented by the layer4_test array, and the true probability distribution is represented by the y_test array. The -np.sum(y_test * np.log(layer4_test),1) expression calculates the cross-entropy loss between the layer4_test and y_test arrays element-wise, and the np.mean() function calculates the mean of these losses.

The result of this code is a scalar value that represents the average cross-entropy loss between the layer4_test and y_test arrays. This value can be used to evaluate the performance of a model on a test set. A lower cross-entropy loss indicates a better fit between the predicted and true probability distributions, which usually translates to better model performance.

def cross_entropy(y_label, y_pred):
    return (-tf.reduce_sum(y_label * tf.math.log(y_pred + 1.e-10)))


optimizer = tf.keras.optimizers.Adam(1e-4)

variables = [W_conv1, b_conv1, W_conv2, b_conv2, 
             W_fc1, b_fc1, W_fc2, b_fc2, ]

def train_step(x, y):
    with tf.GradientTape() as tape:
        current_loss = cross_entropy( y, y_CNN( x ))
        grads = tape.gradient( current_loss , variables )
        optimizer.apply_gradients( zip( grads , variables ) )
        return current_loss.numpy()

In the above code, the cross_entropy() and train_step() functions are defined. These functions are used to train a CNN model on the MNIST dataset.

The cross_entropy() function calculates the cross-entropy loss between the predicted probability distribution and the true probability distribution. The y_label and y_pred arguments represent the true probability distribution and the predicted probability distribution, respectively. The cross-entropy loss is calculated as

-sum(y_label * log(y_pred + 1.e-10))

where y_label and y_pred are tensors with the same shape. The + 1.e-10 term is added to the y_pred tensor to avoid division by zero when calculating the logarithm.

The train_step() function defines a single training step that updates the model’s weights and biases based on the gradient of the cross-entropy loss with respect to the model’s parameters. The x and y arguments represent the input features and labels, respectively. The tf.GradientTape() context manager is used to record the gradient of the cross-entropy loss with respect to the model’s parameters. The current_loss variable is calculated as the cross-entropy loss between the y and y_CNN(x) tensors. The grads variable is calculated as the gradient of current_loss with respect to the model’s parameters. The optimizer.apply_gradients() function is used to apply the calculated gradients to the model’s parameters, which updates the model’s weights and biases. The current_loss value is returned at the end of the function.

These functions will be used to train the CNN model on the MNIST dataset using gradient descent and the Adam optimizer. The Adam optimizer is an optimization algorithm that adapts the learning rate for each parameter based on the first and second moments of the gradients. This can help the model converge faster and with a lower error compared to other optimization algorithms.

correct_prediction = tf.equal(tf.argmax(y_CNN(x_image_train), axis=1), tf.argmax(y_train, axis=1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float32'))

In the above code, the correct_prediction and accuracy variables are defined using TensorFlow operations. These variables are used to evaluate the performance of a trained CNN model on the MNIST dataset.

The correct_prediction variable is a tensor that represents the correct predictions made by the model on the training set. The tf.argmax() function is applied to the output of the y_CNN(x_image_train) function, which returns the indices of the maximum values along the specified axis. In this case, the maximum value is taken along the column axis (axis=1), which corresponds to the predicted class for each example in the batch. The tf.equal() function is then applied to the output of tf.argmax(y_CNN(x_image_train), axis=1) and tf.argmax(y_train, axis=1), which compares the predicted class with the true class for each example in the batch. The result is a tensor of boolean values, which is cast to a tensor of floating-point values using the tf.cast() function.

The accuracy variable is a scalar value that represents the average accuracy of the model on the training set. The tf.reduce_mean() function is applied to the correct_prediction tensor, which calculates the mean of the boolean values contained in the tensor. The result is a scalar value that represents the average accuracy of the model on the training set, where a value of 1.0 indicates that the model made correct predictions for all the examples in the batch, and a value of 0.0 indicates that the model made incorrect predictions for all the examples in the batch.

These variables will be used to evaluate the performance of the trained CNN model on the MNIST dataset. The accuracy measure is a common metric used to evaluate the performance of a classification.

loss_values=[]
accuracies = []
epochs = 1

for i in range(epochs):
    j=0
    # each batch has 50 examples
    for x_train_batch, y_train_batch in train_ds2:
        j+=1
        current_loss = train_step(x_train_batch, y_train_batch)
        if j%50==0: #reporting intermittent batch statistics
            correct_prediction = tf.equal(tf.argmax(y_CNN(x_train_batch), axis=1),
                                  tf.argmax(y_train_batch, axis=1))
            #  accuracy
            accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)).numpy()
            print("epoch ", str(i), "batch", str(j), "loss:", str(current_loss),
                     "accuracy", str(accuracy)) 
            
    current_loss = cross_entropy( y_train, y_CNN( x_image_train )).numpy()
    loss_values.append(current_loss)
    correct_prediction = tf.equal(tf.argmax(y_CNN(x_image_train), axis=1),
                                  tf.argmax(y_train, axis=1))
    #  accuracy
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)).numpy()
    accuracies.append(accuracy)
    print("end of epoch ", str(i), "loss", str(current_loss), "accuracy", str(accuracy) )

In above code, you are training a CNN model on the MNIST dataset using the train_ds2 and train_step functions. The training process consists of several epochs, where an epoch is a complete pass through the training dataset. For each epoch, the training dataset is divided into smaller batches of examples, and the model is trained on each batch using the train_step function.

The train_step function applies the Adam optimization algorithm to the model parameters using the optimizer.apply_gradients() function. The Adam optimization algorithm is a variant of stochastic gradient descent that uses an adaptive learning rate to improve optimization performance. The learning rate is adjusted based on the exponentially decaying average of the past gradients and the past squared gradients.

The train_step function also calculates the current loss value and accuracy for each batch using the cross_entropy() and tf.reduce_mean() functions, respectively. The loss value is calculated as the cross-entropy loss between the true labels and the predicted labels, and the accuracy is calculated as the average of the correct predictions made by the model on the batch.

After each epoch, the final loss value and accuracy for the entire training dataset are calculated using the cross_entropy() and tf.reduce_mean() functions, respectively. The loss values and accuracies for each epoch are then stored in the loss_values and accuracies lists for later analysis.

j=0
acccuracies=[]
# evaluate accuracy by batch and average...reporting every 100th batch
for x_train_batch, y_train_batch in train_ds2:
        j+=1
        correct_prediction = tf.equal(tf.argmax(y_CNN(x_train_batch), axis=1),
                                  tf.argmax(y_train_batch, axis=1))
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)).numpy()
        accuracies.append(accuracy)
        if j%100==0:
            print("batch", str(j), "accuracy", str(accuracy) ) 
import numpy as np
print("accuracy of entire set", str(np.mean(accuracies)))

In above code, you are evaluating the accuracy of the trained CNN model on the MNIST training dataset using the train_ds2 dataset. For each batch of examples in train_ds2, the model’s accuracy is calculated using the tf.reduce_mean() function, which takes the average of the correct predictions made by the model on the batch.

The model’s accuracy is calculated by comparing the model’s predicted labels, which are obtained using the tf.argmax() function on the output of the y_CNN() function, with the true labels, which are obtained using the tf.argmax() function on the y_train_batch tensor. The tf.equal() function is used to determine if the predicted labels and true labels are equal for each example in the batch.

After calculating the accuracy for each batch, the mean accuracy for the entire training dataset is calculated using the np.mean() function on the accuracies list. The mean accuracy is then printed to the console.

It is important to note that this code is only evaluating the model’s accuracy on the training dataset and not on the test dataset. It is generally a good practice to evaluate the model’s performance on a separate test dataset in order to get a more accurate assessment of the model’s generalization ability.

Conclusion

In this code, you have implemented a convolutional neural network (CNN) for classifying handwritten digits from the MNIST dataset. The CNN model consists of two convolutional layers with max pooling, a fully connected layer with dropout, and a final fully connected output layer.

You have also implemented an optimization procedure using the Adam optimizer to train the CNN model on the MNIST training dataset. The model’s performance is evaluated on the training dataset by calculating the mean accuracy on the training examples in each batch and averaging these values over all batches.

It is important to note that this code only evaluates the model’s accuracy on the training dataset and not on the test dataset. It is generally a good practice to evaluate the model’s performance on a separate test dataset in order to get a more accurate assessment of the model’s generalization ability.

A powerful type of Recurrent Neural Networks – Long Short-Term Memory (LSTM) is not only transmitting output information to the next time step, but they are also storing and transmitting the state of the so-called LSTM cell. This cell contains four neural networks – gates that determine which information is stored in the cell state and pushed to output. As a result, the output of the network at a one-time step is dependent on n previous time steps rather than just the previous time step.

In this article, we will look at two similar language modeling problems and see how they can be solved using two different APIs. To begin, we will build a network that can predict words based on the provided text, and we will use TensorFlow for this. In the second implementation, we will use Keras to classify reviews from the IMDB dataset.

Implement a Model with Tensorflow

The DataHandler class in DataHandler.py will be used. This class serves two functions: it loads data from a file and assigns a number to each symbol. The code is as follows:

import numpy as np
import collections

class DataHandler:
    def read_data(self, fname):
        with open(fname) as f:
            content = f.readlines()
        content = [x.strip() for x in content]
        content = [content[i].split() for i in range(len(content))]
        content = np.array(content)
        content = np.reshape(content, [-1, ])
        return content
    
    def build_datasets(self, words):
        count = collections.Counter(words).most_common()
        dictionary = dict()
        for word, _ in count:
            dictionary[word] = len(dictionary)
        reverse_dictionary = dict(zip(dictionary.values(), dictionary.keys()))
        return dictionary, reverse_dictionary

We will create a new class RNNGenerator in rnn_generator.py that can generate an LSTM network based on the parameters passed in.

import tensorflow as tf
from tensorflow.contrib import rnn

class RNNGenerator:
    def create_LSTM(self, inputs, weights, biases, seq_size, num_units):
        # Reshape input to [1, sequence_size] and split it into sequences
        inputs = tf.reshape(inputs, [–1, seq_size])
        inputs = tf.split(inputs, seq_size, 1)
    
        # LSTM with 2 layers
        rnn_model = rnn.MultiRNNCell([rnn.BasicLSTMCell(num_units),rnn.BasicLSTMCell(num_units)])
    
        # Generate prediction
        outputs, states = rnn.static_rnn(rnn_model, inputs, dtype=tf.float32)
    
        return tf.matmul(outputs[–1], weights['out']) + biases['out']

TensorFlow itself, as well as the RNN class from tensorflow.contrib, were imported. We will use our LSTM Network, which is a subtype of RNNs, to create our model. First, we reshaped our input and then divided it into three-symbol sequences. The model was then created.

Using the BasicLSTMCell method, we created two LSTM layers. The parameter num units specify the number of units in each of these layers. Aside from that, we use MultiRNNCell to integrate these two layers into a single network. The static RNN method was then used to build the network and generate predictions.

Finally, we will employ the SessionRunner class in session_runner.py. This class contains the environment used to run and evaluate our model. Here’s how the code works:

import tensorflow as tf
import random
import numpy as np

class SessionRunner():
    training_iters = 50000
            
    def __init__(self, optimizer, accuracy, cost, lstm, initilizer, writer):
        self.optimizer = optimizer
        self.accuracy = accuracy
        self.cost = cost
        self.lstm = lstm
        self.initilizer = initilizer
        self.writer = writer
    
    def run_session(self, x, y, n_input, dictionary, reverse_dictionary, training_data):
        
        with tf.Session() as session:
            session.run(self.initilizer)
            step = 0
            offset = random.randint(0, n_input + 1)
            acc_total = 0
        
            self.writer.add_graph(session.graph)
        
            while step < self.training_iters: if offset > (len(training_data) - n_input - 1):
                    offset = random.randint(0, n_input+1)
        
                sym_in_keys = [ [dictionary[ str(training_data[i])]] for i in range(offset, offset+n_input) ]
                sym_in_keys = np.reshape(np.array(sym_in_keys), [-1, n_input, 1])
        
                sym_out_onehot = np.zeros([len(dictionary)], dtype=float)
                sym_out_onehot[dictionary[str(training_data[offset+n_input])]] = 1.0
                sym_out_onehot = np.reshape(sym_out_onehot,[1,-1])
        
                _, acc, loss, onehot_pred = session.run([self.optimizer, self.accuracy, self.cost, self.lstm], feed_dict={x: sym_in_keys, y: sym_out_onehot})
                acc_total += acc
                
                if (step + 1) % 1000 == 0:
                    print("Iteration = " + str(step + 1) + ", Average Accuracy= " + "{:.2f}%".format(100*acc_total/1000))
                    acc_total = 0
                step += 1
                offset += (n_input+1)

Our model is being run through 50000 iterations. We injected the model, optimizer, loss function, and other information into the constructor so that the class could use it. Naturally, the first step is to slice up the data in the provided dictionary and generate encoded outputs. In addition, we are introducing random sequences into the model to avoid overfitting. The offset variable handles this. Finally, we’ll run the session to determine accuracy. Don’t be confused by the final if statement in the code; it’s just for show (at every 1000 iterations present the average accuracy).

Our main script main.py combines all of this into one, as shown below:

import tensorflow as tf
from DataHandler import DataHandler
from RNN_generator import RNNGenerator
from SessionRunner import SessionRunner

log_path = '/output/tensorflow/'
writer = tf.summary.FileWriter(log_path)

# Load and prepare data
data_handler = DataHandler()

training_data =  data_handler.read_data('meditations.txt')

dictionary, reverse_dictionary = data_handler.build_datasets(training_data)

# TensorFlow Graph input
n_input = 3
n_units = 512

x = tf.placeholder("float", [None, n_input, 1])
y = tf.placeholder("float", [None, len(dictionary)])

# RNN output weights and biases
weights = {
    'out': tf.Variable(tf.random_normal([n_units, len(dictionary)]))
}
biases = {
    'out': tf.Variable(tf.random_normal([len(dictionary)]))
}

rnn_generator = RNNGenerator()
lstm = rnn_generator.create_LSTM(x, weights, biases, n_input, n_units)

# Loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=lstm, labels=y))
optimizer = tf.train.RMSPropOptimizer(learning_rate=0.001).minimize(cost)

# Model evaluation
correct_pred = tf.equal(tf.argmax(lstm,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

# Initializing the variables
initilizer = tf.global_variables_initializer()

session_runner = SessionRunner(optimizer, accuracy, cost, lstm, initilizer, writer)
session_runner.run_session(x, y, n_input, dictionary, reverse_dictionary, training_data)

The content of meditations.txt is “In a sense, people are our proper occupation. Our job is to do them good and put up with them. But when they obstruct our proper tasks, they become irrelevant to us—like sun, wind, and animals. Our actions may be impeded by them, but there can be no impeding our intentions or our dispositions. Because we can accommodate and adapt. The mind adapts and converts to its own purposes the obstacle to our acting. The impediment to action advances action. What stands in the way becomes the way .”

We run the code and get accuracy above 95% with iteration 50000

Implement a model with Keras

This TensorFlow example was straightforward and simple. We used a small amount of data, and the network learned this fairly quickly. What if we have a more complicated issue? Assume we want to categorize the sentiment of each movie review on a website. Fortunately, there is already a dataset dedicated to this issue – The Large Movie Review Dataset (often referred to as the IMDB dataset).

Stanford researchers collected this dataset in 2011. It includes 25000 movie reviews (both positive and negative) for training and the same number of reviews for testing. Our goal is to build a network that can determine which reviews are positive and which are negative.

The power of Keras is that it abstracts a lot of what we had to worry about while using TensorFlow. However, it provides us with less flexibility. Of course, everything has a cost. So, let’s begin by importing the necessary classes and libraries.

There is a slight difference in imports between examples where we used standard ANN and examples where we used Convolutional Neural Network. We brought in Sequential, Dense, and Dropout. Nonetheless, we can see a couple of new imports. We used Embedding and LSTM from keras.layers. As you might expect, LSTM is used to create LSTM layers in networks. In contrast, embedding is used to provide a dense representation of words.

This is an interesting technique for mapping each movie review into a real vector domain. Words are encoded as real-valued vectors in a high-dimensional space, with similarity in meaning corresponding to closeness in the vector space.

We are loading the top 1000 words dataset. Following that, we must divide the dataset and generate and pad sequences. This is accomplished by utilizing sequence from keras.preprocessing

from keras.preprocessing import sequence 
from keras.models import Sequential 
from keras.layers import Dense, Dropout, Embedding, LSTM 
from keras.datasets import imdb 

num_words = 1000 
(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=num_words) 

X_train = sequence.pad_sequences(X_train, maxlen=200) 
X_test = sequence.pad_sequences(X_test, maxlen=200)

# Define network architecture and compile 
model = Sequential() 
model.add(Embedding(num_words, 50, input_length=200)) 
model.add(Dropout(0.2)) 
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2)) 
model.add(Dense(250, activation='relu')) 
model.add(Dropout(0.2)) 
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) 


model.fit(X_train, y_train, batch_size=64, epochs=10) 

print('\nAccuracy: {}'. format(model.evaluate(X_test, y_test)[1]))

We used the number 200 in the padding to indicate that our sequences will be 200 words long. This is how the training input data looks:

Sequential is used for model composition, as we have seen in previous articles. The first layer added to it is Embedding, which we discussed in the previous chapter. We added one LSTM layer after the word embedding was completed. Finally, because this is a classification problem, we add a dense layer with a sigmoid function to determine whether the review was good or bad. Finally, the model is compiled using binary cross-entropy and the Adam optimizer.

We got an accuracy of 85.05%

When developing LSTM networks, we observed two approaches. Both approaches dealt with simple problems and employed a different API. As can be seen, TensorFlow is more detailed and flexible; however, you must take care of a lot more details than when using Keras. Keras is simpler and easier to use, but it lacks the flexibility and possibilities that pure TensorFlow provides. Both of these examples produced acceptable results, but they could have been better. Especially in the second example, where we typically use a combination of CNN and RNN to improve accuracy, but that is a topic for another article.