Speech recognition is artificial intelligence in the form of a virtual assistant that can

be used by giving commands to a computer in the form of sound. In this case, I tried to

develop my own speech recognition system using neural network algorithms.

First, to develop speech recognition, the step that needs to be considered is to choose

the appropriate dataset. The dataset used is a sound sample in the form of audio used as

input. To implement speech recognition also requires algorithms to process datasets and

tested with different parameters so that the accuracy of the algorithms used can produce

maximum results.

The result of this project is to determine how accurate the system is in processing

datasets in the form of sound samples, as well as comparing algorithms between Deep Neural

Network with Convolutional Neural Network

AJI, BAYU CANDRA SENO

Unika Repository

CHAPTER 5IMPLEMENTATION AND RESULTS5.1. ImplementationChapter  5  explains  the  implementation  and testing  of  projects  development  aboutvirtual assistants using Deep Neural Network and Convolutional Neural Network Algorithms.Below is the code of the Deep Neural Network and Convolutional Neural Network algorithmsused to obtain results from the project developed.1. Convolutional Neural NetworkImport necessary modules and dependencies :1. import  os  :  used  to  import  modules,  it  is  a  module  in  python so that  python itselfinteracts directly with the operating system.2. import pathlib :  transfer most file system functions to a path object.3. import matplotlib.pyplot as plt : a collection of functions that make some changes to animage, for example, creating an image, creating a plot areas in an image, adding labelsin the plot and more.4. import  numpy as np :  it  is  one of the python libraries  that  functions for  numericalcomputing processes. NumPy has the ability to create N-dimensional array objects.5. import seaborn as sns : for visualization in this program.18Figure 5.1: Import Library of Convolutional Neural Network Algorithm6. import tensorflow as tf : it is one of the official facilities provided by tensorflow to traindeep learning models, either using a CPU or GPU.7. from tensorflow.keras import layers :  use keras as a library to import layers.8. from tensorflow.keras import models :  use keras as a library to import models.9. from IPython import display : public API for display tools in Ipython.The original  dataset  consists  of  over  105,000 audio files  in  the  WAV (Waveform)audio file format of people saying 35 different words. This data was collected by Google andreleased under a CC BY license. Download and extract the mini_speech_commands.zip filecontaining the smaller Speech Commands Datasets with tf.keras.utils.get_file19Figure 5.2: Import Mini Speech Commands DatasetThe waveforms in the dataset are represented in the time domain. Next, transform thewaveforms  from  the  time-domain  signals  into  the  time-frequency-domain  signals  bycimputing  the  short-time  Fourier  transform  (STFT)  to  convert  the  waveforms  to  asspectrograms, which show frequency changes over time and can be represented as 2D images.Feed the spectrogram images into your neural network to train the model.A Fourier transform (tf.signal.fft) converts a signal to its component frequencies, butloses all time information. In comparison, STFT (tf.signal.stft) splits the signal into windowsof time and runs a Fourier transform on each window, preserving some time information, andreturning a 2D tensor that you can run standard convolutions on.20Figure 5.3: Convert Waveforms to SpectrogramsFor the model,  using a simple Convolutional  Neural  Network (CNN), since it  hastransformed  the  audio  files  into  spectrogram images.  tf.keras.Sequential model  using  thefollowing Keras pre-processing layers:• tf.keras.layers.Resizing : to downsample the input to enable the model to train faster.• tf.keras.layers.Normalization : to normalize each pixel in the image based on its meanand standard deviation.21Figure 5.4: Implementation of Convolutinal Neural NetworkRun the Model on the test set and check the model performance. The main metricsused to evaluate a classification model is accuracy. Accuracy is defined as the percentage ofcorrect predictions for the test data. It can be calculated easily by dividing the number ofcorrect predictions by the number of total predictions.22Figure 5.5: Evaluate The Model Performance2. Deep Neural Network Import necessary modules and dependencies :1. import  os  :  used  to  import  modules,  it  is  a  module  in  python so that  python itselfinteracts directly with the operating system.2. import pathlib :  transfer most file system functions to a path object.3. import matplotlib.pyplot as plt : a collection of functions that make some changes to animage, for example, creating an image, creating a plot areas in an image, adding labelsin the plot and more.4. import  numpy as np :  it  is  one of the python libraries  that  functions for  numericalcomputing processes. NumPy has the ability to create N-dimensional array objects.5. import seaborn as sns : for visualization in this program.6. import tensorflow as tf : it is one of the official facilities provided by tensorflow to traindeep learning models, either using a CPU or GPU.7. from tensorflow.keras import layers :  use keras as a library to import layers.8. from tensorflow.keras import models :  use keras as a library to import models.  9.    from IPython import display : public API for display tools in Ipython.23Figure 5.6: Import Library of Deep Neural Network AlgorithmThe original  dataset  consists  of  over 105,000 audio files  in  the WAV (Waveform)audio file format of people saying 35 different words. This data was collected by Google andreleased under a CC BY license. Download and extract the mini_speech_commands.zip filecontaining the smaller Speech Commands Datasets with tf.keras.utils.get_fileThe waveforms in the dataset are represented in the time domain. Next, transform thewaveforms  from  the  time-domain  signals  into  the  time-frequency-domain  signals  bycimputing  the  short-time  Fourier  transform  (STFT)  to  convert  the  waveforms  to  asspectrograms, which show frequency changes over time and can be represented as 2D images.24Figure 5.7: Import Mini Speech Commands DatasetFigure 5.8: Convert Waveforms to SpectrogramsFeed the spectrogram images into your neural network to train the model. A Fourier transform(tf.signal.fft) converts a signal to its component frequencies, but loses all time information.In the implementation of Deep Neural Network, have different architectures in thelayers. In experiments conducted, Deep Neural Network using 4 layers with filters on eachlayer has a different value.25Figure 5.9: Implementation of Deep Neural NetworkRun the Model on the test set and check the model performance. The main metricsused to evaluate a classification model is accuracy. Accuracy is defined as the percentage ofcorrect predictions for the test data. It can be calculated easily by dividing the number ofcorrect predictions by the number of total predictions.26Figure 5.10: Evaluate The Model Performance5.2. ResultsThe  figure  above  shows  the  loss  function  graph  of  both  algorithms.  On  the  lossfunction graph using parameters that are learning rate 0.001 both Deep Neural Network andConvolutional Network. In Figure 5.11, the loss function graph of the Deep Neural Networkis  explained that  validation loss  on the Deep Neural  Network does  not  decrease even asunstable  so that  the  resulting  accuracy is  less  high.  While  the  loss  function graph of  theConvolutional Neural Network is explained that validation loss on the Convolutional NeuralNetwork is more stable and the graph decreases so that the accuracy generated can be higherthan Deep Neural Network.27Figure 5.11: (a) Deep Neural Network Loss Function Graph and (b) Convolutional NeuralNetwork Loss Function Graph (1)(a) (b)The  figure  above  shows  the  loss  function  graph  of  both  algorithms.  On  the  lossfunction graph using parameters that are learning rate 0.01 both Deep Neural Network andConvolutional Network. In Figure 5.12, the loss function graph of the Deep Neural Networkagain shows a graph that is less stable than before and even the calculation of errors is greaterthan the previous results. While, the loss function graph from Convolutional Neural Networkstill shows a graph that decreases although not as well as the previous one but the calculationof errors that Convolutional Neural Network still remains small.28Figure 5.12: (a) Deep Neural Network Loss Function Graph and (b) Convolutional NeuralNetwork Loss Function Graph (2)(a) (b)The  figure  above  shows  the  loss  function  graph  of  both  algorithms.  On  the  lossfunction graph using parameters that are learning rate 0.1 both Deep Neural Network andConvolutional Network. In Figure 5.13, the loss function graph of both algorithms shows avery different graph than before. So these two algorithms are the same resulting in a badgraph because the graph does not decrease so that the calculation of the resulting error is verylarge and makes accuracy very small.29Figure 5.13: (a) Deep Neural Network Loss Function Graph and (b) Convolutional NeuralNetwork Loss Function Graph (3)(a) (b)Table 5.1: Table Accuracy of  Two Algorithms ResultsAlgorithm Model Parameters AccuracyLearning Rate Epochs Batch SizeDeep Neural NetworkSequential 0.1 10 64 14%Sequential 0.01 10 64 56%Sequential 0.001 10 64 67%Convolutional Neural NetworkSequential 0.1 10 64 12%Sequential 0.01 10 64 81%Sequential 0.001 10 64 82%Table  5.1  shows the  experimental  results  of  both  tested  algorithms with the  threedifferent parameters. From the experiments that have been done, obtained the best accuracyfrom the Deep Neural Network Algorithm of 67% and the accuracy of the ConvolutionalNeural  Network  Algorithm  of  82%.  Judging  from  both  accuracy,  Convolutional  NeuralNetwork  Algorithm are  better  than  Deep  Neural  Network  Algorithm in  terms  of  speechrecognition implementation. From the experiment table above also shows that the learningrate parameters greatly affect the magnitude of accuracy produced, the smaller the learningrate,  the  higher  the  accuracy  result,  while  the  greater  the  learning  rate,  the  smaller  theaccuracy. The success rate of the experiments conducted is measured by the high accuracygenerated, and from the trials conducted by both algorithms resulting in high accuracy.30In the Deep Neural Network Algorithm conducted several experiments with the sameinput shape (124, 129, 1) with 3 different resizing layers namely (16, 16), (32, 32), (64, 64)and described in the 3 tables below:Table 5.2: Model Summary of Deep Neural Network (1)Layer (type) Output Shape Param #resizing (Resizing) (None, 16, 16, 1) 0normalization (Normalization)(None, 16, 16, 1) 3flatten (Flatten) (None, 256) 0dense (Dense) (None, 512) 131584dense (Dense) (None, 256) 131328dense (Dense) (None, 128) 32896dense (Dense) (None, 8) 1032Total params : 296,843Trainable params: 296,840Non-trainable params: 3Test set accuracy: 66%Table 5.3: Model Summary of Deep Neural Network (2)Layer (type) Output Shape Param #resizing (Resizing) (None, 32, 32, 1) 0normalization (Normalization)(None, 32, 32, 1) 3flatten (Flatten) (None, 1024) 0dense (Dense) (None, 512) 524800dense (Dense) (None, 256) 131328dense (Dense) (None, 128) 32896dense (Dense) (None, 8) 1032Total params : 690,059Trainable params: 690,056Non-trainable params: 3Test set accuracy: 67%31Table 5.4: Model Summary of Deep Neural Network (3)Layer (type) Output Shape Param #resizing (Resizing) (None, 64, 64, 1) 0normalization (Normalization)(None, 64, 64, 1) 3flatten (Flatten) (None, 4096) 0dense (Dense) (None, 512) 2097664dense (Dense) (None, 256) 131328dense (Dense) (None, 128) 32896dense (Dense) (None, 8) 1032Total params : 2,262,923Trainable params: 2,262,920Non-trainable params: 3Test set accuracy: 72%The 3 tables above are trials conducted on Deep Neural Network Algorithms with 3different resizing layers. The result of resizing layer used against the resulting accuracy is thatthe greater the resizing layer, the greater the accuracy of the resulting also while the smallerthe resizing layer, the smaller the accuracy of the resulting layer. In the 3 experiments aboveeach using 4 layers with Dense type and 1 layer with Flatten type, and using the same filtersand  activation  function  that  is  relu.  From  the  results  of  the  experiment  above  obtaineddifferent  accuracy  results  from the  3  experiments  conducted.  The  three  experiments  sawimproved accuracy after the resizing layer was changed. All  of the above experiments can be summarized into a conclusion that  indeed toimplement  speech recognition  Convolutional  Neural  Network is  more  suitable  than  DeepNeural  Network  because  datasets  that  were  previously  in  audio  form are  converted  intospectrogram  or  image  data,  in  this  process  Convolutional  Neural  Network  utilizes  theconvolution process by moving a certain-sized convolution kernel (filter) to an image, thecomputer gets new representative information from the results of multiplying the image withthe filter used. While in Deep Neural Network there is no convolution process so the resultingaccuracy is lower than the Convolutional Neural Network.32

VIRTUAL ASSISTANT DEVELOPMENT USING SPEECH RECOGNITION WITH DEEP NEURAL NETWORK ALGORITHM

http://repository.unika.ac.id/28271/6/18.K1.0013-BAYU%20CANDRA%20SENO%20AJI_BAB%20V_a.pdf

VIRTUAL ASSISTANT DEVELOPMENT USING SPEECH RECOGNITION WITH DEEP NEURAL NETWORK ALGORITHM

Abstract

Similar works

Full text

Available Versions

Unika Repository