Skittles Sorter: Technical Details

  1. An amazing video of our project (includes excellent music)

2. Schematic actionSkittles_Schematic

3.  How it all works

Hardware for the skittles sorter included two 180° servos, one 360° servo, a TCS34725 RGB sensor from Adafruit, an RGB LED, an Arduino Uno, and the Atmel A3BU board. The Arduino sent PWM signals to the two 180° servos, which corresponded to angle values between 0° and 180°. The continuous servo also received a PWM value but translated this value to a speed and direction.  The Arduino was incapable of supplying power all components, so the A3BU served as an additional power source.

The star of the show, the RGB sensor (TCS3472), utilized a photodiode array composed of red, green, blue filtered and clear photodiodes, an analog to digital (ADC) converter that converted the photodiode currents to 16 bit values.  The sensor communicated with the Arduino via I2C communication protocol. The Adafruit TCS3472 package also incorporated 3.3V regulator and an onboard LED used to illuminate the target (the skittle). The TCS3472 interfaced with the Arduino by connecting the I2C Clock on the sensor to the SCL (serial clock line) input on the Arduino and the I2C Data on the sensor to the SDA (serial data line) on the Arduino. Adafruit also provided a tutorial and sample code for using the TCS3472.

We planned on using an RGB LED as an indicator light.  We intended for the LED to change color based on the input received from the RGB sensor.  However, mid-project the LED broke and its replacement was not fully functional. Due to time constraints this issue was not investigated further.

The plan for the A3BU was to display a running tab of the number of skittles sorted on its LCD.  We intended to send a digital signal from the Arduino to the A3BU, use this signal to update a variable on the A3BU and display this value on the LCD.  Unfortunately, due to time constraints this sub-project was abandoned. Ultimately, the A3BU was used as an additional power source and to display a simple text phrase on an illuminated LCD.

 

Skittles Sorter: A note on machine learning

When designing the Skittles sorting machine, it seemed fairly obvious that we would need to take controlled sample data on each color of skittle that we were sorting and analyze the RGB results ourselves.

At first we went ahead and gathered the data, loaded it into excel, and ended up with several graphs that looked like so…

The next step would have been to establish lines of regression for the R, G, and B values in relation to the C value for every color of skittle. Then in our code we would have to take the readings and see which color of skittle was most likely being read by applying all 15 of those equations.

That seemed like a lot of work.

So we came up with an easier (and better!) solution. Machine learning.

We plugged our data into a machine learning algorithm that generated a decision tree for us that looked something like (well a lot like) this.

There was no automatic way to transfer this decision tree into our code, so we had to type it out ourselves (more on that in a bit).

Now, it may seem odd that the decisionmaking process was based on relatively few, hardcoded, discrete, value tests when the RGB values behaved according to continuous relationships. The machine learning algorithm seemed to notice that even though the RGB values varied with brightness (the C value), they tended to fall within certain ‘bands’ for each skittle color since the brightness tended to stay within a reasonable range (due to the builtin chip LED).

We translated this tree into code by typing out the if-else statements ourselves. The translated version of the tree pictured above turned out like this.

The actual machine learning algorithm used was pulled from a python library. It is pasted at the bottom of this post.

Within the code, we typed out our datapoints (can be plainly seen in the beginning section). Each array within the larger array is one datapoint. The ‘labels’ array tells the algorithm which category each datapoint falls under in sequence (each color is associated with a number); thankfully we typed out our data in uniform sections. The ‘feature names’ array gives aliases to each value in a datapoint, and the ‘class names’ array translates from the numeric category in the ‘labels’ array to a name like ‘Red’ or ‘Blue’.

from sklearn import tree

features = [
    [1740, 926, 744, 3049], # Red
    [1886, 318, 386, 2150],
    [1987, 1126, 964, 3862],
    [2226, 1077, 1029, 3932],
    [1801, 375, 436, 2132],
    [1522, 306, 356, 1776],
    [1611, 411, 427, 2042],
    [1723, 501, 505, 2274],
    [1626, 399, 424, 1997],
    [1617, 411, 430, 2048],
    [6073, 1410, 1282, 7949], # Orange
    [3743, 2115, 1558, 7304],
    [4135, 853, 675, 5256],
    [3578, 1139, 850, 5297],
    [4901, 1029, 811, 6338],
    [3430, 1860, 1293, 6562],
    [4362, 875, 698, 5611],
    [2938, 735, 568, 3982],
    [4096, 1200, 826, 5772],
    [4254, 2015, 1493, 7470],
    [3243, 3032, 1499, 8010], # Yellow
    [3150, 2880, 1288, 7540],
    [4148, 2887, 1035, 8054],
    [6079, 5697, 2387, 14570],
    [3778, 2928, 1025, 7648],
    [5326, 3857, 1311, 10652],
    [4609, 3200, 1071, 8958],
    [3883, 3004, 1132, 7940],
    [4340, 2969, 1013, 8285],
    [1849, 1772, 997, 4551],
    [2557, 3800, 1820, 7581], # Green
    [1171, 1510, 599, 2881],
    [1193, 1576, 587, 3085],
    [1572, 2067, 1083, 4241],
    [2035, 2771, 1451, 6236],
    [1837, 2315, 1340, 5459],
    [1309, 1734, 696, 3316],
    [1019, 1223, 520, 2396],
    [1058, 946, 743, 2575],
    [2540, 3732, 1831, 7464],
    [438, 195, 183, 585], # Purple
    [521, 314, 278, 842],
    [655, 521, 435, 1310],
    [476, 303, 263, 837],
    [525, 309, 273, 851],
    [1252, 759, 849, 2385],
    [678, 559, 476, 1414],
    [460, 232, 215, 690],
    [1223, 891, 933, 2649],
    [955, 559, 630, 1743],
    [977, 862, 619, 2460], # Nothing
    [977, 862, 619, 2460],
    [972, 858, 616, 2449],
    [972, 858, 616, 2449],
    [974, 860, 618, 2452],
    [976, 863, 620, 2461],
    [974, 861, 619, 2459],
    [973, 860, 619, 2452],
    [973, 860, 618, 2453],
    [974, 860, 618, 2457],
]
labels = [
    0,0,0,0,0,0,0,0,0,0,
    1,1,1,1,1,1,1,1,1,1,
    2,2,2,2,2,2,2,2,2,2,
    3,3,3,3,3,3,3,3,3,3,
    4,4,4,4,4,4,4,4,4,4,
    5,5,5,5,5,5,5,5,5,5
]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(features, labels)

# viz code
from sklearn.externals.six import StringIO
import pydot

dot_data = StringIO()
tree.export_graphviz(clf,
                     out_file=dot_data,
                     feature_names=["R", "G", "B", "C"],
                     class_names=["Red", "Orange", "Yellow", "Green", "Purple", "Nothing"],
                     filled=True, rounded=True,
                     impurity=False)
graph = pydot.graph_from_dot_data(dot_data.getvalue())
graph.write_pdf("tree.pdf")</pre>

Skittles Sorter: Overview

Skittle Sorter

Our final project is a Skittles Sorting Machine. The machine uses a RGB color sensor, a RGB LED, three servos, an Arduino, and the A3BU AVR microprocessor controller. This machine was constructed with K’Nex, cardboard, electrical tape, duct tape, scotch tape, super glue, a nike shoe box, modeling clay, and rubber bands. We also ordered some 3-D printed parts. Most of our electrical parts came from the lab group’s personal inventory, however a few parts were ordered through school using Quartzy.  The Arduino IDE 1.6.8 was used to program the Arduino and Atmel Studio was used to program the A3BU. Eagle CAD was used to draw the electrical schematic.
First, we assembled the structure to hold the servos, Arduino, breadboard, A3BU, and tubes used to move the skittles through the automation process. Next, all the electrical parts were connected and the Arduino was programmed to control the servos and sensor. Finally, the process was tested and RGB values were obtained for various colors so we could program the sorting servo to swing into the correct position.
The project was successful because we used ideas and principles learned in class to construct an automated sorting machine. The machine was able to take in a bag of unsorted skittles, put them through the system, and come out the other end sorted into correct piles.