Symmetric Positive Definite (SPD) matrix learning methods have become popular in many image and video processing tasks, thanks to their ability to learn appropriate statistical representations while respecting Riemannian geometry of un- derlying SPD manifolds. In this paper we build a Riemannian network architecture to open up a new direction of SPD ma- trix non-linear learning in a deep model. In particular, we de- vise bilinear mapping layers to transform input SPD matrices to more desirable SPD matrices, exploit eigenvalue rectifi- cation layers to apply a non-linear activation function to the new SPD matrices, and design an eigenvalue logarithm layer to perform Riemannian computing on the resulting SPD ma- trices for regular output layers. For training the proposed deep network, we exploit a new backpropagation with a variant of stochastic gradient descent on Stiefel manifolds to update the structured connection weights and the involved SPD matrix data. We show through experiments that the proposed SPD matrix network can be simply trained and outperform exist- ing SPD matrix learning and state-of-the-art methods in three typical visual classification tasks.