Deeply-Supervised Nets


Deeply-supervised nets (DSN) is a method that simultaneously minimizes classification error and improves the directness and transparency of the hidden layer learning process. Three aspects of convolutional-neural-network-type (CNN-type) architectures are being addressed: (1) transparency in the effect intermediate layers have on overall classification; (2) discriminativeness and robustness of learned features, especially in early layers; (3) training effectiveness in the face of "vanishing" gradients.  To combat these issues, "companion" objective functions are introduced at each hidden layer, in addition to the overall objective function at the output layer. The advantage of such integrated deep supervision is evident: (1) for small training data and relatively shallower networks, deep supervision functions as a strong “regularization” for classification accuracy and learned features; (2) for large training data and deeper networks deep supervision makes it convenient to exploit the significant performance gains that extremely deep networks can bring by improving otherwise problematic convergence behavior. For more details or other related projects, please visit http://www.cogsci.ucsd.edu/~ztu