Dimensionality Reduction

Abstract

Principal Component Analysis (PCA) and linear autoencoders (LAEs) both perform linear dimensionality reduction by minimizing squared reconstruction error, and the Baldi-Hornik theorem establishes their theoretical equivalence at global optima [1]. However, the practical conditions under which gradient-based autoencoder training recovers the PCA subspace remain underexplored. This study systematically evaluates the architectural and optimization requirements for subspace convergence using the MNIST handwritten digit dataset. We compare PCA and LAEs across three experimental conditions: (1) unconstrained architecture with bias terms and independent encoder-decoder weights, (2) constrained architecture satisfying Baldi-Hornik conditions (tied weights, no bias), and (3) optimized hyperparameter configurations on the full 70,000-sample training set. Subspace alignment is quantified using principal angles and Grassmann distance, and our results demonstrate that the architectural constraints are essential: unconstrained linear autoencoders achieve mean principal angles of 21-50 degrees relative to PCA, despite identical reconstruction errors (MSE difference < 4%), and constrained linear autoencoders reduce angles to 2.7-4.5 degrees, representing a 6-11x improvement. With optimized hyperparameters—specifically cosine learning rate scheduling, 200 training epochs, and orthogonal initialization—constrained autoencoders achieve near-perfect alignment with mean angles as low as 0.12 degrees for latent dimension k=128. Hyperparameter analysis reveals that learning rate scheduling is the dominant factor (30x impact), followed by learning rate magnitude (5x impact), with initialization showing negligible effect. These findings confirm that the Baldi-Hornik equivalence holds empirically under appropriate architectural constraints and optimization configurations, clarifying when linear autoencoders faithfully recover the PCA subspace in practice.