This paper introduces the Principal Orthogonal Complement Thresholding (POET) method for estimating high-dimensional covariance matrices with conditional sparsity and fast-diverging eigenvalues. The method assumes a sparse error covariance matrix in an approximate factor model, allowing for cross-sectional correlation even after removing common but unobservable factors. The POET estimator includes the sample covariance matrix, factor-based covariance matrix, thresholding estimator, and adaptive thresholding estimator as specific cases. The paper provides mathematical insights into the relationship between principal component analysis and factor analysis in high-dimensional data. It studies the convergence rates of the sparse residual covariance matrix and the conditional sparse covariance matrix under various norms, showing that the impact of estimating unknown factors vanishes as dimensionality increases. The paper also derives uniform convergence rates for unobserved factors and their factor loadings, and verifies asymptotic results through simulation studies. A real data application on portfolio allocation is presented. The POET estimator is defined by thresholding the remaining components of the sample covariance matrix after removing the first K principal components. The method is optimization-free and computationally appealing. The paper shows that the POET estimator is equivalent to a least squares substitution estimator and provides asymptotic properties under various norms. The paper also discusses the estimation of the idiosyncratic covariance matrix and its precision matrix, showing that the POET estimator achieves optimal convergence rates under certain conditions. The results demonstrate that the POET estimator can consistently estimate the covariance matrix and its inverse in high-dimensional factor models.This paper introduces the Principal Orthogonal Complement Thresholding (POET) method for estimating high-dimensional covariance matrices with conditional sparsity and fast-diverging eigenvalues. The method assumes a sparse error covariance matrix in an approximate factor model, allowing for cross-sectional correlation even after removing common but unobservable factors. The POET estimator includes the sample covariance matrix, factor-based covariance matrix, thresholding estimator, and adaptive thresholding estimator as specific cases. The paper provides mathematical insights into the relationship between principal component analysis and factor analysis in high-dimensional data. It studies the convergence rates of the sparse residual covariance matrix and the conditional sparse covariance matrix under various norms, showing that the impact of estimating unknown factors vanishes as dimensionality increases. The paper also derives uniform convergence rates for unobserved factors and their factor loadings, and verifies asymptotic results through simulation studies. A real data application on portfolio allocation is presented. The POET estimator is defined by thresholding the remaining components of the sample covariance matrix after removing the first K principal components. The method is optimization-free and computationally appealing. The paper shows that the POET estimator is equivalent to a least squares substitution estimator and provides asymptotic properties under various norms. The paper also discusses the estimation of the idiosyncratic covariance matrix and its precision matrix, showing that the POET estimator achieves optimal convergence rates under certain conditions. The results demonstrate that the POET estimator can consistently estimate the covariance matrix and its inverse in high-dimensional factor models.