TY - GEN
T1 - Improving gradient-based learning algorithms for large scale feedforward networks
AU - Ventresca, M.
AU - Tizhoosh, H. R.
PY - 2009
Y1 - 2009
N2 - Large scale neural networks have many hundreds or thousands of parameters (weights and biases) to learn, and as a result tend to have very long training times. Small scale networks can be trained quickly by using second-order information, but these fail for large architectures due to high computational cost. Other approaches employ local search strategies, which also add to the computational cost. In this paper we present a simple method, based on opposite transfer functions which greatly improve the convergence rate and accuracy of gradientbased learning algorithms. We use two variants of the backpropagation algorithm and common benchmark data to highlight the improvements. We find statistically significant improvements in both converegence speed and accuracy.
AB - Large scale neural networks have many hundreds or thousands of parameters (weights and biases) to learn, and as a result tend to have very long training times. Small scale networks can be trained quickly by using second-order information, but these fail for large architectures due to high computational cost. Other approaches employ local search strategies, which also add to the computational cost. In this paper we present a simple method, based on opposite transfer functions which greatly improve the convergence rate and accuracy of gradientbased learning algorithms. We use two variants of the backpropagation algorithm and common benchmark data to highlight the improvements. We find statistically significant improvements in both converegence speed and accuracy.
KW - Backpropgation
KW - Gradient-based learning
KW - Large scale networks
KW - Opposite transfer functions
KW - Opposition-based computing
UR - http://www.scopus.com/inward/record.url?scp=70449455565&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=70449455565&partnerID=8YFLogxK
U2 - 10.1109/IJCNN.2009.5178798
DO - 10.1109/IJCNN.2009.5178798
M3 - Conference contribution
AN - SCOPUS:70449455565
SN - 9781424435531
T3 - Proceedings of the International Joint Conference on Neural Networks
SP - 3212
EP - 3219
BT - 2009 International Joint Conference on Neural Networks, IJCNN 2009
T2 - 2009 International Joint Conference on Neural Networks, IJCNN 2009
Y2 - 14 June 2009 through 19 June 2009
ER -