The project research the hypothesis that the GeLU function gives good performance on image classification problems because of the property of not resetting the derivatives of the negative values close to zero in the backward process of the neural network.
The reason for testing this feature is that the GeLU function is complicated to implement and expensive in terms of hardware, so the project dealt with the use of this feature on a computationally simpler activation function, so the solution was to replace the GeLU function with a function that is similar to relu, which is much simpler, with the addition of the feature mentioned above.
The results were surprisingly good, this feature is really what improves the performance and that the new implementation passed the ReLU function and got closer to the GeLU function.
The conclusion from this is that not resetting the derivatives of the negative values close to zero in the backward process in the neural network helps the learning process of the network.