Reading is one of the most complex skills that we utilize daily, and it involves the early development and interaction of various lower-level subskills, including phonological processing and oral language. These subskills recruit brain structures, which begin to develop long before the skill manifests and exhibit rapid development during infancy. However, how longitudinal trajectories of early brain development in these structures support long-term acquisition of literacy subskills and subsequent reading is unclear. Children underwent structural and diffusion MRI scanning at multiple timepoints between infancy and second grade and were tested for literacy subskills in preschool and decoding and word reading in early elementary school. We developed and implemented a reproducible pipeline to generate longitudinal trajectories of early brain development. We then examined whether these trajectories were associated with literacy (sub)skills or influenced by familial risk of reading difficulty and children’s home literacy environments, two common literacy-related covariates. Results showed that individual differences in curve features (e.g., intercepts and slopes) for longitudinal trajectories of volumetric, surface-based, and white matter organization measures were linked directly to phonological processing and indirectly to early elementary school decoding and word reading skills via phonological processing. Altogether, these findings suggest that the brain bases of phonological processing, previously identified as one of the strongest behavioral predictors of decoding and word reading skills, may already begin to develop by birth but undergo further refinement between infancy and preschool. The present study underscores the importance of considering academic skill acquisition from the very beginning of life.