In the past, I wrote a couple of blog entries on the theory of deep learning. I plan to continue when I have some more free time.
Example blog 1: Training invariances in deep nets and its consequences I: Set up and Problem description
Example blog 2: Training invariances in deep nets and its consequences II: Linear models