This paper presents a method to invert image representations, including SIFT, HOG, and deep convolutional neural networks (CNNs), by solving a regularized regression problem. The approach involves finding an image whose representation best matches a given code, using a combination of a loss function and a regularizer that captures natural image properties. The method is evaluated on both shallow and deep representations, showing that it can reconstruct images more accurately than existing techniques. For shallow representations like HOG and DSIFT, the method achieves better results than recent alternatives, highlighting differences in their invertibility. For deep CNNs, the method reveals that layers progressively build more invariance, with the final layers capturing abstract and photometrically accurate information. The study also shows that CNNs retain detailed visual information, even in their deeper layers, and that different layers capture varying degrees of geometric and photometric invariance. The method is applied to analyze the information stored in CNNs, revealing how they progressively build more abstract representations of images. The results demonstrate that CNNs can be inverted effectively, with the final layers producing reconstructions that are visually meaningful, even though they capture only a sketch-like representation of the original image. The study also shows that CNNs have strong non-natural confounders, which can affect the reconstruction process. Overall, the method provides insights into the properties of image representations and their ability to capture visual information at different levels of abstraction.This paper presents a method to invert image representations, including SIFT, HOG, and deep convolutional neural networks (CNNs), by solving a regularized regression problem. The approach involves finding an image whose representation best matches a given code, using a combination of a loss function and a regularizer that captures natural image properties. The method is evaluated on both shallow and deep representations, showing that it can reconstruct images more accurately than existing techniques. For shallow representations like HOG and DSIFT, the method achieves better results than recent alternatives, highlighting differences in their invertibility. For deep CNNs, the method reveals that layers progressively build more invariance, with the final layers capturing abstract and photometrically accurate information. The study also shows that CNNs retain detailed visual information, even in their deeper layers, and that different layers capture varying degrees of geometric and photometric invariance. The method is applied to analyze the information stored in CNNs, revealing how they progressively build more abstract representations of images. The results demonstrate that CNNs can be inverted effectively, with the final layers producing reconstructions that are visually meaningful, even though they capture only a sketch-like representation of the original image. The study also shows that CNNs have strong non-natural confounders, which can affect the reconstruction process. Overall, the method provides insights into the properties of image representations and their ability to capture visual information at different levels of abstraction.