A cutting-edge unsupervised method for noise removal, dimensionality reduction, anomaly detection, and more
All the tutorials about TensorFlow and neural networks I have shared until now have been about supervised learning. This one will be about the Autoenocder which is an unsupervised learning technique. If I want to express it simply, autoencoders reduce the noises from the data by compressing the input data, and encoding and reconstructing the data. That way autoencoders can reduce the dimensionality or the noise of the data and focus on the real focal point of the input data.
As you can see from the introduction to the autoencoders here there is more than one process required.
- First, a model to compress the input data which is the encoder model.
- Then another model to reconstruct the compressed data that should be as close as the input data which is a decoder model.
In this process, it can remove the noise, reduce the dimensionality, and clear up the input data.
In this tutorial, I will explain in detail how an autoencoder works with a working example.
For this example, I chose to use a public dataset (Apache License 2.0) named deep_weeds.
import tensorflow as tf
import tensorflow_datasets as tfds
ds = tfds.load('deep_weeds', split='train', shuffle_files=True)
Data Preparation
We need to prepare a dataset for this unsupervised anomaly detection example. Only one class will be taken as our main class that will be considered as the valid class. And I will put a few data from another class as an anomaly. Then we will develop the model to see if we can find that few anomaly data.
I chose class 5 as the valid class and class 1 as the anomaly. In the code block below, I am taking all the data of classes 5 and 1 first and creating lists of the images and their corresponding labels.
import numpy as np
images_main = ()
images_anomaly = ()
labels_main= ()
labels_anomaly = ()
ds = ds.prefetch(tf.data.AUTOTUNE)
for example in ds…
