This paper investigates the problem of semi-supervised image classification and image clustering. Unlike pre- vious methods to develop sophisticated classifier models, our method learns a new image representation from all available data (labeled and unlabeled) by exploiting the patterns of data distribution in a novel manner. Particularly, a rich set of visual prototypes are sampled from all available data, and are taken as surrogate classes to train discriminative classifiers; images are projected (classified) via the classifiers and the projected values (similarities to the prototypes) are stacked to build a feature vector. The training set is noisy. Hence, in the spirit of ensemble learning we create a set of such training sets which are all diverse, leading to di- verse classifiers. The method is dubbed Ensemble Projection (EP). EP captures not only the characteristics of indi- vidual images, but also the relationships among images. It is conceptually simple and computationally efficient, yet ef- fective and flexible. Experiments on eight standard datasets show that: (1) EP outperforms previous methods for semi- supervised image classification; (2) EP produces promising results for self-taught image classification, where unlabeled samples are a random collection of images rather than being from the same distribution as the labeled ones; and (3) EP improves over the original features for image clustering. The code of the method is available at www.vision.ee. ethz.ch/ Ìdaid/EnProDeepFets.