Sign-Interfaced Machine Operating Network (SIMON)

The purpose of Sign-Interfaced Machine Operating Network, or SIMON, is to translate sign language into numbers. We used a machine learning classifier to classify images of hands into the corresponding symbols or actions. SIMON interprets American Sign Language and displays the translation on the LCD screen through the following steps. The given image of the ASL sign was extracted to produce a single numerical digit using a machine learning model. That value was then serialized to ASCII and sent to the microcontroller. The microcontroller would deserialize the value and post the result to the LCD screen. Hardware used to complete this project included the ATMega328P Xplained Mini Board, a laptop with Windows, a webcam, and an LCD screen. The goal of final project was achieved as we were successfully able to read the American Sign Language images and display the desired result on the LCD screen.

SIMON Sign Language Translator Deep Learning Image Processing Code

Below is one of the deep learning training methods in the Python code running on the Windows machine to analyze and classify images. Please see the comments for a detailed description of the program.

	
def train(self, learning_rate = 0.0001, num_epochs = 1500, minibatch_size = 32, print_cost = True):
"""
Created By: Jacob Taylor Cassady
Last Updated: 2/7/2018
Objective: Implements a three-layer tensorflow neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SOFTMAX.
Arguments:
learning_rate -- learning rate of the optimization
num_epochs -- number of epochs of the optimization loop
minibatch_size -- size of a minibatch
print_cost -- True to print the cost every 100 epochs		
Returns:
parameters -- parameters learnt by the model. They can then be used to predict.
"""
print("Entering train....")
ops.reset_default_graph()
(n_x, m) = self.train_matrix.shape    # (n_x: input size, m : number of examples in the train set)
n_y = self.train_targets.shape[0]		  # n_y : output size.
costs = []							  # Used to keep track of varying costs.

# Create placeholders for TensorFlow graph of shape (n_x, n_y)
print("Creating placeholders for TensorFlow graph...")
X, Y = self.create_placeholders(n_x, n_y)
print("Complete.\n")

# Initialize Parameters
print("Initailizing parameters for TensorFlow graph...")
parameters = self.initialize_parameters()
print("Complete.\n")

# Build the forward propagation in the TensorFlow Graph
print("Building the forward propagation in the TensorFlow Graph...")
Z3 = self.forward_propagation(X, parameters)
print("Complete.\n")

# Add the cost function to the Tensorflow Graph
print("Adding cost function to the TensorFlow Graph")
cost = self.compute_cost(Z3, Y)
print("Complete.\n")

# Define the TensorFlow Optimizer.. We are using an AdamOptimizer.
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

# Initialize all the variables with our newly made TensorFlow Graph
init = tf.global_variables_initializer()

# Use the TensorFlow Graph to train the parameters.
with tf.Session() as session:
# Run the initialization
session.run(init)

# Perform Training
for epoch in range(num_epochs):
	epoch_cost = 0.								# Defines a cost related to the current epoch
	num_minibatches = int(m / minibatch_size)	# Calculates the number of minibatches in the trainset given a minibatch size
	minibatches = random_mini_batches(self.train_matrix, self.train_targets, minibatch_size)

	for minibatch in minibatches:
		# Retrieve train_matrix and train_targets from minibatch
		mini_matrix, mini_targets = minibatch

		# Run the session to execute the "optimizer" and the "cost",
		_, minibatch_cost = session.run([optimizer, cost], feed_dict={X:mini_matrix, Y:mini_targets})

		# Sum epoch cost
		epoch_cost += minibatch_cost / num_minibatches

	# Done training.  Print the cost of every 100 epochs
	if print_cost == True and epoch % 100 == 0:
		print("Cost after epoch %i: %f" % (epoch, epoch_cost))
	# Keep track of the cost of every 5 epochs for plotting later
	if print_cost == True and epoch % 5 == 0:
		costs.append(epoch_cost)

# Plot the costs for analysis
plt.plot(np.squeeze(costs))
plt.ylabel('cost')
plt.xlabel('iteration ( per 5 )')
plt.title('Learning rate = ' + str(learning_rate))
if print_cost == True:
	#plt.show()
	pass

# Save the parameters as a varaible for prediction and evaluation of fit to test set.
parameters = session.run(parameters)

# Develop TensorFlow prediction standards for testing accuracy  of test and train sets
correct_prediction = tf.equal(tf.argmax(Z3), tf.argmax(Y))

# Develop accuracy identifier using TensorFlow
accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))

# Display accuracy of train and test predictions.
print("Train Accuracy: ", accuracy.eval({X: self.train_matrix, Y: self.train_targets}))
print("Test Accuracy: ", accuracy.eval({X: self.test_matrix, Y: self.test_targets}))

# Return parameters for prediction against the model.
self.parameters = parameters

train_accuracy = accuracy.eval({X: self.train_matrix, Y: self.train_targets})
test_accuracy = accuracy.eval({X: self.test_matrix, Y: self.test_targets})

accuracies = {"train_accuracy": train_accuracy,
              "test_accuracy" : test_accuracy}

# Return parameters for prediction against the model.
return parameters, accuracies