Research has repeatedly shown that early adversities are a risk factor for psychological distress in adulthood. However, limited research has examined the association between early adversities and intraindividual changes in psychological distress among middle-aged adults. This study developed a self-report scale to retrospectively measure the level of adversity experienced in childhood (Early Environment Scale: EES) and investigated the association between the EES score and the change in psychological distress in middle-aged adults. Study 1 examined the reliability and validity of the nine-item EES using a sample of 337 Japanese adults (162 women; Meanage = 42.96, SDage = 10.55). In Study 2, a three-wave six-month longitudinal survey was conducted with a sample of 2000 Japanese adults (1000 women; Meanage = 40.94; SDage = 5.35). Psychological distress at all three waves was measured with the Kessler Psychological Distress Scale (K-6). The results from a latent growth analysis showed significant positive associations between early adversities and the intercept and slope factors of psychological distress. Specifically, individuals who reported more early adversity showed higher initial levels of psychological distress and a stronger increase in psychological distress among Japanese middle-aged adults. Early adversities might influence the development of dispositional sensitivity for psychological distress in adulthood.