Theories of Enactivism propose an action-oriented approach to understand human cognition. So far, however, empirical evidence supporting these theories has been sparse. Here, we investigate whether spatial navigation based on allocentric reference frames that are independent of the observer’s physical body can be understood within an action-oriented approach. Therefore, we performed three experiments testing the knowledge of the absolute orientation of houses and streets towards north, the relative orientation of two houses and two streets, respectively, and the location of houses towards each other in a pointing task. Our results demonstrate that under time pressure, the relative orientation of two houses can be retrieved more accurately than the absolute orientation of single houses. With infinite time for cognitive reasoning, the performance of the task using house stimuli increased greatly for the absolute orientation and surpassed the slightly improved performance in the relative orientation task. In contrast, with streets as stimuli participants performed under time pressure better in the absolute orientation task. Overall, pointing from one house to another house yielded the best performance. This suggests, first, that orientation and location information about houses are primarily coded in house-to-house relations, whereas cardinal information is deduced via cognitive reasoning. Second, orientation information for streets is preferentially coded in absolute orientations. Thus, our results suggest that spatial information about house and street orientation is coded differently and that house orientation and location is primarily learned in an action-oriented way, which is in line with an enactive framework for human cognition.