The ANN was designed to fit the UCI Breast Cancer dataset. We followed the following rationale while developing this model:
• Implementing a feed-forward multilayer architecture with configurable neurons, layers, activation functions, loss function, learning rate and schedule, and type of gradient descent.
• Testing different gradient descent algorithms, weights, biases, learning rates, and hidden layers, to record the best performance and classification accuracy rates.
• Ensuring the code is easy to debug and structure, with appropriate comments for users. Methods
The program was written using the Python coding language and developed on Jupyter notebook. The Breast Cancer dataset is then explored and cleaned before splitting it into training and testing subsets. Feature scaling is then performed to normalize the range of variables in the dataset and to speed up the training. This included, normalizing the X training and testing sets with min-max where the minimum value of a feature is made a 0 and the maximum value is made a 1, and all other values become floats between 0 and 1.

The hyperparameters are initialized and made configurable according to the coursework specifications. Users can choose between sigmoid, ReLU, tanh and softmax activation functions, batch, mini-batch and stochastic gradient descent algorithms, the learning rate, number of input features/nodes and hidden neurons/nodes and layers, epochs, and loss functions. Different values were tested to report the highest accuracy rates and recorded in this paper. The weights and biases of all the layers in the network were randomly initialized and stored in one dictionary.

Subsequently, for forward propagation, the linear activation value and the output of the layer is calculated by using the activation function given for that layer. All the values are stored in lists and are used during backward propagation to retrieve the values whenever required. Later, the loss is calculated and averaged during mini-batch or stochastic gradient descent and the gradients computed during the backward pass are used to update the weights and biases. This happens for the set number of epochs/iterations.

1e-5 was added to the output probability obtaining log (0) i.e., negative infinity that arises from using the cross-entropy loss function, and to avoid division by zeroes when calculating the gradient of the first activation value in the backward propagation. The decision threshold value was set to 0.39 and was calculated by plotting the ROC Curve when running an MLP-Classifier from scikitlearn with parameters used for our model and adjusted according to trial and error with the hyperparameters. The learning rate decay used when enabled, is the Step Decay schedule that drops the learning rate by half every 10 epochs.

The Loss Function
Apart from the cross-entropy loss function, two other loss functions were selected for our model that were suitable for binary classification. A literature review was done as follows -
The Hinge Loss function was developed for training classifiers, primarily Support Vector Machine (SVM), and is typically used in binary classification. According to Rennie and Srebro, ‘An important property of this loss function is that it is an upper bound on the zero-one misclassification error’. Thus, it is used for "maximum-margin" classification.
Hinge loss has reportedly performed better than cross-entropy loss in certain situations concerning binary classification problems and can be used in the context of neural networks.
An extension of this function is the Squared Hinge Loss, which is simply the square of the hinge loss score. Thus, we found it appropriate to choose the following additional loss functions for our model.

Results
The following inferences were made after testing different hyperparameters over our model:
• Using all the input features gave the best accuracy.
• The cross-entropy loss function has the highest convergence rate and accuracy values.
• The hinge and squared hinge loss functions give increased accuracies, but the costs remain stagnant at the values 0.8-0.04, therefore the convergence rate to 0 is reduced.
• Increasing the number of epochs, while reducing the batch size and learning rate increased accuracy.
• Learning rates between 0.00095 and 0.01 perform well for all hyperparameters.
• Stochastic gradient descent takes longer to converge, as compared to batch and minibatch gradient descents but usually records the highest accuracy with fewer epochs and a lesser learning rate compared to mini-batch and batch algorithms.
• Increasing the hidden layers led to reduced accuracy after increasing until 3 layers.
• The stochastic and mini-batch gradient descents show the highest accuracies and lowest costs, with accuracy values ranging from 90-95%. Batch gives similar accuracies with an increased number of epochs.
• The step learning decay schedule shows optimum results (increased accuracies and reduced costs) with mini-batch gradient descent. It outperforms time-based or exponential decays schedules.
• The ReLU activation function works the best compared to tanh activation function for the hidden layers of the model. A combination of ReLU activation functions on each hidden layer with sigmoid as the output layer’s activation function should be configured for getting the right probabilities.
• The mean results across a series of 10 runs for the hyperparameters with a learning rate of 0.03 and no learning rate decay, 4 layers, each with 25, 21, 59 and 1 nodes, and activation functions ReLU, ReLU, tanh, and sigmoid respectively, cross-entropy loss function, batch gradient descent for 1000 epochs is 93%. The standard deviation is 0.0179. The node numbers were chosen to be set at 2/3rd the size of the input layer + size of the output layer as recommended (Krishnan, 2022).
• The highest accuracies recorded with a lower number of nodes, a combination of 3 hidden layers each with nodes 8, 8, and 1, cross-entropy loss function, ReLU, tanh, and sigmoid (output layer) activation functions, 1000 epochs, and a learning rate of 0.003 with no decay, and batch gradient descent is 95%. The loss converges to a rate of 0.06, indicating a well-trained model.

Discussion
Our dataset influenced our data preprocessing steps, such as MinMax normalization, lower learning rates, threshold values, and hidden neuron numbers, and thus dictated the structure of our MLP model. Due to the limited number of samples available, the model was at risk of overfitting, so a simple ANN structure was optimal for implementation. The train-test set was divided into 75% - 25% rows and was evaluated for performance. A literature review for suitable loss functions was done, and hinge and squared hinge loss proved appropriate for binary classification. The binary cross-entropy loss function gave the highest convergence rate and accuracy values. For each hyperparameter value investigated, the training and testing process is performed multiple times to record accurate inferences and to account for the inherent random variation in training by averaging the results.

Through thorough experimentation of the hyperparameters, we discovered the right balance of values that worked well with our model. Graphs and figures were charted subsequently to reflect these results. Further work that can be considered to fine-tune our model is utilizing softmax activation function and two output nodes in the output layer for better performance. The softmax activation functions and its derivative were implemented for this reason but fall short of complete implementation.
This project was highly informative and led to a well-rounded understanding of MLP Neural Networks. Additional functionalities were implemented as we progressed in our understanding and knowledge of this program.