The purpose of Sign-Interfaced Machine Operating Network, or SIMON, is to translate sign language into numbers. We used a machine learning classifier to classify images of hands into the corresponding symbols or actions. SIMON interprets American Sign Language and displays the translation on the LCD screen through the following steps. The given image of the ASL sign was extracted to produce a single numerical digit using a machine learning model. That value was then serialized to ASCII and sent to the microcontroller. The microcontroller would deserialize the value and post the result to the LCD screen. Hardware used to complete this project included the ATMega328P Xplained Mini Board, a laptop with Windows, a webcam, and an LCD screen. The goal of final project was achieved as we were successfully able to read the American Sign Language images and display the desired result on the LCD screen.

# Tag: Team Simon

## SIMON Sign Language Translator Schematics

Schematic representing SIMON deep neural network used to categorize images:

Schematic and picture of SIMON Xplained Mini board and LCD display module in final configuration:

Flowchart diagram for high-level SIMON product description:

## SIMON Sign Language Translator Deep Learning Image Processing Code

Below is one of the deep learning training methods in the Python code running on the Windows machine to analyze and classify images. Please see the comments for a detailed description of the program.

def train(self, learning_rate = 0.0001, num_epochs = 1500, minibatch_size = 32, print_cost = True): """ Created By: Jacob Taylor Cassady Last Updated: 2/7/2018 Objective: Implements a three-layer tensorflow neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SOFTMAX. Arguments: learning_rate -- learning rate of the optimization num_epochs -- number of epochs of the optimization loop minibatch_size -- size of a minibatch print_cost -- True to print the cost every 100 epochs Returns: parameters -- parameters learnt by the model. They can then be used to predict. """ print("Entering train....") ops.reset_default_graph() (n_x, m) = self.train_matrix.shape # (n_x: input size, m : number of examples in the train set) n_y = self.train_targets.shape[0] # n_y : output size. costs = [] # Used to keep track of varying costs. # Create placeholders for TensorFlow graph of shape (n_x, n_y) print("Creating placeholders for TensorFlow graph...") X, Y = self.create_placeholders(n_x, n_y) print("Complete.\n") # Initialize Parameters print("Initailizing parameters for TensorFlow graph...") parameters = self.initialize_parameters() print("Complete.\n") # Build the forward propagation in the TensorFlow Graph print("Building the forward propagation in the TensorFlow Graph...") Z3 = self.forward_propagation(X, parameters) print("Complete.\n") # Add the cost function to the Tensorflow Graph print("Adding cost function to the TensorFlow Graph") cost = self.compute_cost(Z3, Y) print("Complete.\n") # Define the TensorFlow Optimizer.. We are using an AdamOptimizer. optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) # Initialize all the variables with our newly made TensorFlow Graph init = tf.global_variables_initializer() # Use the TensorFlow Graph to train the parameters. with tf.Session() as session: # Run the initialization session.run(init) # Perform Training for epoch in range(num_epochs): epoch_cost = 0. # Defines a cost related to the current epoch num_minibatches = int(m / minibatch_size) # Calculates the number of minibatches in the trainset given a minibatch size minibatches = random_mini_batches(self.train_matrix, self.train_targets, minibatch_size) for minibatch in minibatches: # Retrieve train_matrix and train_targets from minibatch mini_matrix, mini_targets = minibatch # Run the session to execute the "optimizer" and the "cost", _, minibatch_cost = session.run([optimizer, cost], feed_dict={X:mini_matrix, Y:mini_targets}) # Sum epoch cost epoch_cost += minibatch_cost / num_minibatches # Done training. Print the cost of every 100 epochs if print_cost == True and epoch % 100 == 0: print("Cost after epoch %i: %f" % (epoch, epoch_cost)) # Keep track of the cost of every 5 epochs for plotting later if print_cost == True and epoch % 5 == 0: costs.append(epoch_cost) # Plot the costs for analysis plt.plot(np.squeeze(costs)) plt.ylabel('cost') plt.xlabel('iteration ( per 5 )') plt.title('Learning rate = ' + str(learning_rate)) if print_cost == True: #plt.show() pass # Save the parameters as a varaible for prediction and evaluation of fit to test set. parameters = session.run(parameters) # Develop TensorFlow prediction standards for testing accuracy of test and train sets correct_prediction = tf.equal(tf.argmax(Z3), tf.argmax(Y)) # Develop accuracy identifier using TensorFlow accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float')) # Display accuracy of train and test predictions. print("Train Accuracy: ", accuracy.eval({X: self.train_matrix, Y: self.train_targets})) print("Test Accuracy: ", accuracy.eval({X: self.test_matrix, Y: self.test_targets})) # Return parameters for prediction against the model. self.parameters = parameters train_accuracy = accuracy.eval({X: self.train_matrix, Y: self.train_targets}) test_accuracy = accuracy.eval({X: self.test_matrix, Y: self.test_targets}) accuracies = {"train_accuracy": train_accuracy, "test_accuracy" : test_accuracy} # Return parameters for prediction against the model. return parameters, accuracies