Creating a Simple Neural Network with Keras, TensorFlow, Python and SQL Server

Greetings, friends.

Welcome to this edition of our newsletter, where we're diving into the techy side of AI. This time, we're exploring how to build a simple neural network using Keras, TensorFlow, Python, and Microsoft SQL Server as our database that holds the training data. It's worth noting that this journey into the depths of neural networks and the examination of alternative approaches will unfold in a comprehensive and detailed article. Brace yourself for a deep dive into the world of AI, as this promises to be an extensive exploration.

About Keras

Keras, is an open-source deep learning framework written in Python. It serves as a high-level neural networks API, allowing for easy and fast prototyping of deep learning models. Originally developed as an independent project, Keras has become an integral part of the TensorFlow library, starting with TensorFlow version 2.0.

About TensorFlow

TensorFlow is an open-source machine learning framework developed by the Google Brain team. It was initially released by Google in 2015 and has since become one of the most widely used and popular tools in the field of machine learning and deep learning.

TensorFlow provides a comprehensive platform for building and deploying machine learning models, offering a flexible architecture that supports various hardware configurations, including CPUs, GPUs, and TPUs (Tensor Processing Units). Its versatility makes it suitable for a wide range of applications, from simple linear regression to complex deep neural networks.

About Python

Python is a widely-used, user-friendly programming language renowned for its simplicity and extensive libraries. In the realm of AI development, Python's popularity has surged due to its ease of use and robust support for machine learning and deep learning frameworks such as TensorFlow and PyTorch.

This versatility makes it a go-to choice for AI practitioners, facilitating the implementation of complex algorithms and the creation of sophisticated models. Its widespread adoption in the AI community aligns well with your diverse technology interests, offering a powerful toolset for integrating AI capabilities into your projects and staying at the forefront of advancements in the field.

About SQL Server

SQL Server is a Relational Database Management System (RDBMS) developed by Microsoft, renowned for its robustness and scalability in handling vast datasets. I have been working with SQL Server for over two decades and for this reason, I wanted to include it in this project since I find it very interesting to combine the technological power of Python and SQL Server. Therefore, I used SQL Server for storing the Iris dataset training data. At this point, it is noteworthy to mention SQL Server's Machine Learning Services, a powerful feature that integrates seamlessly with the database engine. This allows you to perform advanced analytics and machine learning directly within the SQL Server environment, offering a comprehensive solution for data processing and model building.

Nevertheless, for this project, I have not utilized these services since the purpose of this experimentation was to develop my AI model and neural network using Keras and Tensorflow as the model development frameworks, along with using SQL Server as the platform onto which the Iris dataset was stored. Perhaps we'll talk about SQL Server Machine Learning Services in a future edition of the newsletter.

The Dataset

In this project, I used the Iris dataset. This dataset was introduced by the British statistician and biologist Ronald Fisher in his 1936 paper “The use of multiple measurements in taxonomic problems”. This dataset is a well-known benchmark, featuring measurements of iris flowers that include sepal length, sepal width, petal length, and petal width along with providing the species for each variation. In total, the dataset presents 3 species (aka “classes”) for the different variations of its 4 features.

The Iris dataset served as the foundation for our neural network exploration, where we harnessed the power of Keras, TensorFlow, Python, and SQL Server. Through this combination, our model was trained to decipher patterns and make predictions based on the intricate details encapsulated in the Iris dataset. The elegance and richness of the Iris dataset make it an ideal choice for unraveling the intricacies of neural network development.

Why using a Neural Network for this Project?

While various options exist for making predictions with the Iris dataset, I opted for a distinctive approach by choosing to build and train a neural network. Although traditional machine learning algorithms, statistical models, and ensemble methods are commonly employed for datasets of similar size, my deliberate decision to delve into the intricacies of neural networks sets this project apart. The focus was not merely on achieving predictions but on examining the process of constructing and training a neural network using Keras, TensorFlow, Python, and SQL Server. This choice reflects a commitment to exploring cutting-edge techniques and showcasing the versatility of neural networks even in scenarios with modest data dimensions.

Architectural Design

Below, I provide the architectural design of my simple AI model/neural network:

Let’s discuss more about the components that interact between them in this design, in order to give working AI neural network.

• SQL Server: The SQL Server database contains the iris_data table, which stores the training data for the neural network, including features (sepal_length, sepal_width, petal_length, petal_width) and the corresponding species.
• Input Layer: Represents the input features of the neural network, corresponding to the columns in the iris_data table. As you can see, I’m using 4 features in total.
• Hidden Layer (Dense - ReLU): This is the hidden layer of my neural network with 64 neurons and the Rectified Linear Unit (ReLU) activation function. This layer processes the input features. I’m using the ReLU activation function since it introduces non-linearity to the neural network, enabling efficient learning of complex patterns in the Iris dataset, while also addressing issues like the vanishing gradient problem and promoting sparsity for improved feature learning.
• Output Layer (Dense - Softmax): This is the output layer with 3 neurons, employing the Softmax activation function for multi-class classification. It produces the predicted probabilities for each class.
• Model Training and Compilation: This is the process that involves the compilation and training of the neural network using the specified architecture, optimizer (Adam), and loss function (sparse categorical cross-entropy).
• Model Prediction: After training, the model can be used to make predictions on new data.

Setting up the Development Environment

As discussed above, the development environment for this project consists of:

• SQL Server (learn more)
• Python (learn more)
• TensorFlow (learn more)
• Keras (learn more)
• Visual Studio Code (learn more)

Moreover, I needed to install the following libraries:

• Pyodbc (learn more)
• Pandas (learn more)
• scikit-learn (learn more)

Also, note that I have created the development environment in a machine with Windows OS and I have not used GPU processing since this is a very simple AI model.

So, after installing SQL Server and creating the “IrisDB” database, I have created a table named “iris_data”.

Then, I populated the "iris_data" table with all the records of the Iris dataset, that is 150 records in total. This table, contains the 4 features of the flowers (sepal_length, sepal_width,petal_length, petal_width, and species). So, in simple words, it contains the observations/combinations of the 4 features and the corresponding species each time which can be one of the below:

• “setosa” species
• “versicolor” species
• “virginica” species

Then I needed to install Python. At this point, note that for installing Python in order to be used with TensorFlow and Keras, you need to take into consideration the supported versions.

At the time of creating this project, Keras required Python 3.9 or later.

Also, TensorFlow required Python 3.9 – 3.11.

So based on the two above requirements, I decided to install Python 3.11.5 for my Windows environment. Note that during Python installation, I selected the option to add python.exe to the path (run as administrator).

In case you are reproducing this project on your own development environment, make sure that you are using the latest Python version supported by Keras and TensorFlow.

After installing Python, I needed to install TensorFlow following the instructions at: https://www.tensorflow.org/install/source_windows

To this end, I installed the perquisites using pip, i.e.:

pip3 install -U six numpy wheel packaging        

pip3 install -U keras_preprocessing --no-deps        

Also, I made sure I had installed the Visual C++ build tools 2019.

Then I installed TensorFlow, using pip and by following the on-screen instructions along with verifying the installation:

python -m pip install tensorflow        

python -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"        

?

After TensorFlow, I installed Keras:

pip install --upgrade keras        

?

And after Keras, I installed Pyodbc, Pandas and scikit-learn, again using pip:

pip install pyodbc        

pip install pandas        

pip install -U scikit-learn        

Finally, I installed Visual Studio Code along with (optionally) adding the official Microsoft extension for Python.

As you can see, there are many perquisites for getting started, but once you install everything in order along with its prerequisites, the process will be smooth.

Building the AI Model – The Code

So, now that the development environment is ready along with all its prerequisites and the dataset is also ready in our SQL Server database, let’s talk about the code. Below, I will be presenting the different blocks of my code and will be explaining each block.

Import Libraries and Frameworks:

import pyodbc
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder        

In the above block of my code, I import necessary libraries and frameworks for the project.

• pyodbc: For connecting to SQL Server.
• pandas: For working with data in a tabular format.
• Sequential and Dense from keras.models: For building a sequential neural network.
• train_test_split from sklearn.model_selection: For splitting the dataset into training and testing sets.
• LabelEncoder from sklearn.preprocessing: For encoding categorical labels.

Trusted Connection to SQL Server 2022 Named Instance:

conn_str =  "DRIVER={ODBC Driver 17 for SQL Server};SERVER=.\SQL2K22;DATABASE=IrisDB;Trusted_Connection=yes;"
conn = pyodbc.connect(conn_str)        

With the above code, I establish a connection to a SQL Server database named "IrisDB" on a local named instance "SQL2K22" using Windows authentication.

Now, let's see the actual steps in my code that have to do with the model creation, training and validation.

Step 1: Retrieve Training Data from the "iris_data" table:

training_query = 'SELECT sepal_length, sepal_width, petal_length, petal_width, species FROM iris_data'
training_data = pd.read_sql(training_query, conn)        

In this step, I retrieve the training data from the "iris_data" table in the "IrisDB" database using the SQL query provided. The data is loaded into a Pandas DataFrame (training_data).

Step 2: Prepare training data:

X_train = training_data[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']]
y_train = training_data['species']
label_encoder = LabelEncoder()
y_train_encoded = label_encoder.fit_transform(y_train)        

In this step, I prepare the training data by separating features (X_train) and labels (y_train). In this case, the label is the "species" value in each record. Moreover, I encode the categorical labels using LabelEncoder.

Step 3: Build and Train Keras Model:

model = Sequential()
model.add(Dense(units=64, input_dim=4, activation='relu'))
model.add(Dense(units=3, activation='softmax'))  # Assuming 3 iris species

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train_encoded, epochs=10, batch_size=32)        

This is the critical step, where I define and train a simple neural network model using Keras. The model has one hidden layer with 64 units and an output layer with 3 units (assuming 3 iris species). The Adam optimizer and sparse categorical cross-entropy loss are used. Training is done for 10 epochs with a batch size of 32.

Close the connection before proceeding:

conn.close()        

In this code block, I close the connection to the SQL Server database after retrieving the training data and training the model.

Step 4: Generate New Prediction Data (Simulating Unseen Data):

new_prediction_data = pd.DataFrame({
    'sepal_length': [5.1, 6.2, 4.8],
    'sepal_width': [3.5, 3.0, 3.8],
    'petal_length': [1.4, 4.5, 1.6],
    'petal_width': [0.2, 1.5, 0.2],
    'species': ['setosa', 'versicolor', 'setosa']  # Add actual species for accuracy calculation
})        

In this step, I create a new DataFrame (new_prediction_data) to simulate unseen data and for validating my model for the purpose of this example. In practice, you would have a separate dataset for predictions. The actual species names are only included for accuracy calculation.

Display the input data for predictions:

print()
print('Input Data for Predictions:')
print(new_prediction_data)
print()        

In this code block, I just print on screen the input data for predictions, providing visibility into the characteristics of the new data.

Step 5: Make Model Predictions on New Prediction Data:

prediction_features = new_prediction_data[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']]
new_predictions = model.predict(prediction_features)

predicted_labels = label_encoder.inverse_transform(new_predictions.argmax(axis=1))        

In this step, I use the trained model to make predictions on the new prediction data. The predicted labels are decoded from numerical values (i.e. class numbers) back to species names.

Calculate accuracy:

correct_predictions = (predicted_labels == new_prediction_data['species']).sum()
total_predictions = len(new_prediction_data)
accuracy = correct_predictions / total_predictions * 100        

In this code block, I calculate the accuracy of the predictions by comparing the predicted labels to the actual labels in the new prediction data.

Display the predictions and accuracy:

print('\nPredicted Labels for the given values:')
print(predicted_labels)
print('\nPrediction Accuracy: {:.2f}%'.format(accuracy))
print()        

In this code block, I print on screen the predicted labels and the accuracy of the predictions on the new data. The accuracy is displayed as a percentage.

Validating the AI Model

So, now that our project is ready, we can run the code in order to train our model with the 150 records of the Iris Dataset and then let it predict the species for the given prediction data defined in step 4 in the code above and which are also shown below.

    'sepal_length': [5.1, 6.2, 4.8],
    'sepal_width': [3.5, 3.0, 3.8],
    'petal_length': [1.4, 4.5, 1.6],
    'petal_width': [0.2, 1.5, 0.2]        

This is the output/result that we get for this specific prediction data:

In this specific run, the model correctly predicted the species for 2 out of the 3 instances in the new prediction data as it correctly predicted twice the "setosa" species and missed correctly predicting the versicolor species for the given prediction data. Therefore, in this example and for the given input data and our AI model's configuration parameters, the accuracy of prediction was 66.67%.

So, by analyzing the model's validation results we can see that there is room for improvement. Note that, fine-tuning hyperparameters, including the number of hidden units, layers, and epochs, can significantly impact model performance, since exploring different combinations may lead to improved results.

To this end, I further tuned the training "epochs" parameter and changed its value from 10 to 20 and then revalidated the model:

As you can see, the change of the "epochs" parameter didn't improve the model's accuracy.

So I went ahead and fine-tuned again the "epochs" parameter by changing its value from 20 to 30 and revalidated the model once more:

In this case, as you can see, the model's accuracy has been indeed improved and it is now a perfect 100%.

So we can see that, increasing the number of training epochs from 10 to 30 has proven to be a pivotal adjustment, leading to a significant improvement in the model's accuracy, achieving a perfect 100%. The decision to extend the training duration allows the neural network to undergo more iterations over the entire dataset, enabling it to better capture complex patterns and relationships within the training data. In essence, the model benefits from an extended learning period, refining its internal parameters and representations to align more closely with the underlying patterns of the Iris dataset.

This adjustment is particularly effective when the initial training period might have been insufficient for the model to converge to an optimal state. It underscores the importance of finding an appropriate balance in the number of epochs during model training, where too few may result in underfitting, and an adequate number allows the model to better generalize to new, unseen data. The increased accuracy highlights the responsiveness of the model to extended training, showcasing the impact of optimizing this hyperparameter on its overall performance.

Strategies/Best Practices to Improve Model Performance

As a general guideline, to improve model performance, consider the following strategies/best practices:

• Data Augmentation: Expand the dataset with more diverse and representative samples to enhance the model's ability to generalize.
• Feature Engineering: Explore creating new features or transforming existing ones to provide the model with more relevant information.
• Model Architecture: Experiment with different neural network architectures, adjusting layers, units, and activation functions to find an optimal structure.
• Hyperparameter Tuning: Systematically search for optimal hyperparameter values, including learning rate, batch size, and the number of training epochs.
• Regularization: Apply techniques like dropout or L2 regularization to prevent overfitting and improve generalization.
• Cross-Validation: Use cross-validation techniques during training to assess how well the model generalizes to different subsets of the data.
• Evaluate on Test Set: Reserve a portion of the dataset as a test set to evaluate the model's performance on truly unseen data.
• Ensemble Methods: Explore combining predictions from multiple models using ensemble methods to enhance overall performance.
• Normalization and Scaling: Ensure proper normalization and scaling of input features for improved neural network performance.
• Monitor and Debug: Regularly monitor the training process, evaluate model performance, and adjust strategies if the model is not improving as expected.

These strategies, when iteratively applied and tailored to the specific characteristics of the data, can contribute to the continuous refinement and optimization of the neural network model.

Conclusion

In conclusion, the journey of creating a neural network with Keras, TensorFlow, Python, and SQL Server for Iris flower species prediction has provided valuable insights.

The model demonstrated its ability to learn and generalize from the provided dataset, achieving an initial accuracy of 66.67%. By carefully analyzing the findings and employing strategic improvements, such optimizing hyperparameters, the model's accuracy was ultimately elevated to an impressive 100% by extending the number of training epochs to 30.

This success underscores the importance of a thoughtful and iterative approach in developing machine learning models. It emphasizes the significance of continuous experimentation, fine-tuning, and monitoring to achieve optimal performance.

The journey serves as a testament to the dynamic nature of neural network development, where thoughtful adjustments lead to remarkable enhancements.

At this point, I would like to express my gratitude for being a part of the GnoelixiAI Hub Newsletter and engaging in the exciting journey of exploring neural network development and the world of AI with me. Your interest and enthusiasm are truly appreciated.

As a token of my appreciation, I'll be sharing the code from my project on my GitHub repository, allowing you to access and explore the details of the neural network implementation. Additionally, I'm excited to announce that a video walkthrough of the project will be soon available on GnoelixiAI Hub's YouTube channel. This video will provide a visual guide through the code and offer insights into the strategies employed for optimizing model performance. The link for my GitHub repository will be provided in the video's description when published.

To stay up-to-date with the latest developments, code releases, and video tutorials, be sure to also subscribe to GnoelixiAI Hub's YouTube channel and check often GnoelixiAI Hub's website.

Thank you once again for being a part of this collaborative learning experience!

Best Regards,

Artemakis

Additional Resources:

• My interview (in Greek) on the podcast “Town People” in “Old Town Radio”, where we discussed Artificial Intelligence.
• Download the AI QuickStart - Cheat sheet on GnoelixiAI Hub.

Read Also:

• AI QuickStart: Your Essential Cheat Sheet
• Evolution of Programming Practices: From Manual to AI
• Achieving Human-AI Synergy via Collaborative Intelligence
• Innovations for Tomorrow: AI and Citizen Well-Being
• Large Language Models (LLMs) and their Impact on Everyday Technology
• Democratizing AI Education: Exploring the Potential of Explainable AI (XAI)
• The Importance of Explainable AI (XAI)