We propose a novel three-layered approach for semantic segmentation of building facades. In the first layer, starting from an over-segmentation of a facade, we employ the recently introduced machinelearning technique Recursive Neural Networks (RNN) to obtain a probabilistic interpretation of each segment. In the middle layer, initial labeling is augmented with the information coming from specialized facade component detectors. The information is merged using a Markov Random Field defined over the image. In the highest layer, we introduce weak architectural knowledge, which enforces the final reconstruction to be architecturally plausible and consistent. Rigorous tests performed on two existing datasets of building facades demonstrate that we significantly outperform the current-state of the art, even when using outputs from lower layers of the pipeline. In the end, we show how the output of the highest layer can be used to create a procedural reconstruction.