Context plays an important role in IoT enabled smart environment as it allows more meaningful interpretation of collected data. Currently, Deep Learning approaches, in particular Deep Neural Networks (DNNs), have gained attention as they are suitable for big data. DNNs have shown their potential in various applications by learning the complex mappings between input and output data. However, their applicability is limited in the domains where the mappings are influence by context information. During the secondment learning mechanisms will be further investigated to realize a Contextual Deep Learning framework.