We propose a novel method for weakly supervised se- mantic segmentation. Training images are labeled only by the classes they contain, not by their location in the image. On test images instead, the method predicts a class label for every pixel. Our main innovation is a multi-image model (MIM) - a graphical model for recovering the pixel labels of the training images. The model connects superpixels from all training images in a data-driven fashion, based on their appearance similarity. For generalizing to new test images we integrate them into MIM using a learned multiple kernel metric, instead of learning conventional classifiers on the recovered pixel labels. We also introduce an âobjectnessâ potential, that helps separating objects (e.g. car, dog, hu- man) from background classes (e.g. grass, sky, road). In ex- periments on the MSRC 21 dataset and the LabelMe subset of , our technique outperforms previous weakly super- vised methods and achieves accuracy comparable with fully supervised methods.