Perceived spatial presence and body orientation affect the recall of out-of-sight places in an immersive sketching experiment

In the judgment of relative direction (JRD) task, subjects are asked to imagine a known environment from a certain point of view, heading towards some remembered object within this environment. From the imagined heading thus defined, they are then asked to report the relative direction to other remembered objects (Shelton & McNamara, 2001). Recall fidelity depends on the imagined heading direction and is best if it aligns with an intrinsic axis of the imagined environment, in the experiment the long axis of a rectangular room. The intrinsic axis may also be defined by objects placed inside the room. Mou and McNamara (2002) placed several objects in an regular grid and found better JRD performance for mental tasks that involved perspectives that were aligned with one of the two grid axes. On a larger scale, the layout of streets in a city or the corridors in an office building may also provide intrinsic axes of reference (Montello, 1991; Werner & Schmidt, 1999). Reference axes for imagery of spaces on different scales such as an office space within an university campus may coexist and need not be identical (Wang & Brockmole, 2003).

Performance in the JRD task also depends on the subject’s body orientation during recall and is superior if their actual body orientation is aligned with the imagined body orientation or heading (May, 2004; Kelly et al., 2007; Riecke & McNamara, 2017). This “sensorimotor alignment effect” indicates that despite their engagement in imagery and the JRD task, subjects still maintain a sense of their actual bodily position as might be obtained from path integration and spatial updating. This is true even if both the “actual” and the imagined body pose are defined in a virtual environment only (Marchette et al., 2014). The representational space generated by imagery of a remote environment thus combines references derived from remembered cues of the imagined space itself (the intrinsic axis) and directional tracking or spatial updating between the imagined and the actual observer position (sensorimotor alignment; for overview, see Julian et al. 2018, Meilinger & Vosgerau, 2010).

Representational memory is organized along preferred reference axes even if no such axes are explicitly induced by the instruction given to the subject. Basten et al. (2012) have shown that the priming of the recall orientation by imagined travel influences the imagery of distant places. In this study, subjects located on an university campus have been asked to imagine walking in the down-town between two familiar city squares, thereby crossing the target square in one of two possible directions. When subsequently asked to produce a sketch map of the target square, the direction of the previously imagined travel primed the recall orientation in the sense that subjects were more likely to sketch the target square as is would have appeared during the imagined walk. The authors conclude that performing an imagined walk activates the representational memory due to automated spatial updating or mental travel or both.

In the Basten et al. (2012) study, priming is generated by the activation of visual imagery during imagined travel and may result from a visual memory. Alternatively, it may be a result of a spatial updating process in which memories of nearby locations are automatically activated as soon as the observer approaches that target location. Röhrich et al. (2014) therefore asked passers-by in the city center to produce sketch maps of nearby (walking distance) city squares which were out-of-sight from the interview location. Sketch maps were rated for orientation and showed significant variation with interview location. Sketch maps of nearby locations are predominantly aligned with the airline direction from the recall position to the target places as if subjects could look through the intermittent buildings. Subjects interviewed at more distant locations (about 2 km away) produced more homogeneous maps showing a standard or “canonical” view of the target independent of the interview location. Here, we use the term “canonical” in analogy of the canonical views known from object recognition (Bülthoff et al., 1995). The imagery of the distant target square thus depends on two directions, first, the airline direction, i.e., on how the square would look could we see through the intermittent buildings, and second, an intrinsic axis of the target square defining its canonical view.

In addition to these dependencies, the bodily orientation of the subjects might also play a role. This was demonstrated by Meilinger et al. (2016) using a building task instead of freely sketched maps. Subjects were given a set of cards naming popular locations around town and asked to rebuild the configuration. The study confirms the position dependence of the recall orientation and additionally demonstrates an effect of body orientation. Note, however, that the Meilinger et al. (2016) study uses landmark buildings in a larger area, not views of a single city square. The building task was transferred to a virtual environment by Le Vinh et al. (2020), again with the reconstruction of a specific target square as a task. The study confirmed the overall effect of airline direction; body orientation was not addressed.

Unlike object configurations in a room that can be perceived at a single glance, larger areas such as the down-town area of a city have to be explored in a step-by-step fashion, a distinction that has been discussed as vista and navigational spaces by Montello (1993). For the resulting representation, Meilinger (2008) suggested a network of local reference frames or charts, each with its own intrinsic axis. In a simple JRD experiment, only one such chart will be activated and imagery will be based on this chart’s intrinsic axis and the body orientation of the observer. In position-dependent recall, however, the actual and the imagined environments are usually part of different local charts each with their own intrinsic axis and the relative position of these charts will affect imagery as an additional factor. In the studies discussed above, this relative position is described by the airline direction and the distance to the target. In the present study, we present novel data on the role of body orientation in position-dependent recall.

Except for the Le Vinh et al. (2020) study, all studies on position dependent recall discussed above are based on real-world experiments. With the present paper, we also want to validate the VR technology as a tool allowing better experimental control. After decades of discussion, it is now generally assumed that virtual environments can provide valid experimental results in spatial cognition, both from non-immersive desk-top setups (e.g., Ruddle & Jones, 2001) and with motion tracking and head-mounted goggles (e.g., Avraamides & Kelly, 2008; Kelly et al., 2007; Marchette et al., 2014). While the fidelity of visual input does not seem to play a major role in the perception of presence as defined by Slater et al. (2009), it may still be relevant for navigation or the recognition of particular locations in a virtual environment (Lessels & Ruddle, 2005). This latter aspect is crucial for our study; we therefore use the term “spatial presence” to denote the sense of being at a particular location rather than somewhere else. Spatial presence is one dimension assessed in the presence questionnaire of Sanchez-Vives and Slater (2005). The other dimensions, immersion and the memories built from VR experience, are not addressed in our study. Besides the visual fidelity the quality of the feedback from the subjects’ bodily motion may compromise the validity of VR experimentation, especially in non-immersive setups (Kelly et al., 2018).

One additional problem arising in the transfer of the position-dependent recall paradigm in a virtual environment is the task to be performed by the subjects. Le Vinh et al. (2020) used an interactive building task in which a joystick was used to grab building blocks from a reservoir and place them on a work space. Here, we explore a novel approach called immersive sketching, in which the subject uses an ordinary drawing board while still immersed in the virtual environment. The sketching process is recorded with a camera and overlaid to the VR simulation as a live image.

Here, we set out to test the hypothesis that position-dependent recall occurs also in virtual environments and that in this case the relevant “position” is the participants’ simulated position in the virtual environment, not their physical position in the lab. In addition, we access the influence of body orientation. The experiment uses the novel “immersive sketching” paradigm in which participants produce sketch maps of remote places while immersed in a virtual environment. In a second experiment, we addressed the question whether the imaginary view point assumed during the immersion phase persists after leaving the virtual environment or is replaced by a direction corresponding to the participant’s physical position.

A total of 100 healthy adults, 37 male and 63 female, participated in our studies. All participants were students of the University of Tübingen aged at least 18 years. Further details were not recorded. All included subjects had been living in Tübingen for more than 2 years and successfully passed a pre-experiment testing their knowledge of the experimental area. Eighty participants were randomly assigned to 4 groups of 20 people each for experiment one and 2 groups of 10 people each for experiment two. Before the study, all the participants confirmed that they had no problem with virtual reality and wearing an Oculus RIFT head-mounted display.

Participants were informed that they could stop and quit the study any time during the experiment without providing any reason. All participants gave written informed consent and received a payment of 8 Euro for their participation.

Virtual environments were presented with an Oculus RIFT Development Kit 2 (DK2) head-mounted display with a total horizontal field of view of \(100^{\circ }\) and \(960 \times 1080\) pixel resolution for each eye. During the presentation, subjects were seated on a rotating chair, as shown in Fig. 1. The chair has been prepared for the experiments in the following way:

We have used square pieces of paper for the experiments, so that the subject is not biased for drawing horizontally or vertically. As the piece of paper was fixed with a clip to the desk, it was not possible for the subjects to move or rotate the paper while sketching.

Videos for the pre-experiment were captured on site in Tübingen using a Canon Power Shot G7 camera. They were presented in the Oculus RIFT, which in this case was used just as a simple displaying device, i.e., in open loop.

For the main experiments, we have set up a virtual environment in Unity using C#. For each sketching location (S1, S2, S3), we created a 360-degree panorama of the location by taking 12 pictures at 30\(^{\circ }\) intervals with the Canon Power Shot G7 camera mounted on a tripod. The 12 pictures were stitched to a cylindrical panorama using Microsoft Image Composite Editor. The three panoramas are shown in Fig. 2. All pictures and videos were taken during early morning hours to avoid imaging of passers-by.

The sketching locations were modeled by placing the panoramas as textures in the inside of a virtual cylinder. For the top of the cylinder, we have used matching images of skies, for example blue sunny sky or cloudy sky. For the bottom of the cylinder, we have chosen images of a matching ground, for example cobblestone.

For the immersive sketching procedure, the live picture of the sketching paper fixed on the tablet arm desk is displayed in a square work space (virtual sketching board) appearing in the bottom part of the cylinder. This allows the subjects to draw a sketch map on the paper without taking off the goggles. The texture of this square is set to Unity’s WebCamTexture that takes and displays the camera’s live image on the virtual sketching board instead of a static texture or plain color. The participants thus see their drawing hand and the produced sketch while still immersed in the VE. The work space for sketching is placed either towards or away from the target locations, such that subjects have to produce their sketches while bodily oriented towards or away from the target square.

The subject’s point of view in the experiments was placed in the center of the cylinder. The Oculus RIFT allowed for accurate tracking of the movement of the subject’s head, so that the subjects could visually explore the virtual environment. We have only tracked the rotation of the head; translational movement produced with the upper body while seated was not fed back to the VR simulation, i.e., if the subjects did produce translational movements, the cylinder would virtually move with them and the image remained unchanged.

During the experiment, the experimenter could switch between the three sketching locations using the space bar of a laptop computer, thereby “teleporting” the subjects from one location to another.

In all experiments, we used three sketching locations (S1, S2, S3) and two target locations (T1: market square, T2: timber market); see Figs. 2 and 3. All locations were well-known places in the historic center of Tübingen. All locations were in walking distance of each other but mutually out-of-sight.

Each of the two target locations was sketched from two different directions provided by the two closest sketching locations. This results in the four sketching tasks S1-T1 (at S1, sketch T1), S2-T1, S2-T2, and S3-T2; see Fig. 3. In addition, we varied the subjects’ body orientation by placing the live image of the sketching paper either towards the street leading from the sketching location directly to the target (“towards condition”) or offset from this direction by 180\(^{\circ }\) (“away condition”). In the “towards” condition, subjects might therefore imagine to just move forward to walk to the target. The “exit direction” is generally similar to the airline direction, but slight deviations may occur due to the city street raster. In the “away” condition, subjects were required to produce their sketches while imagining a target located behind them.

In total, these variations result in eight different tasks. To avoid interactions between repeated sketching tasks, each subject produced two sketches only.

The maps have been categorized by two raters according to the cardinal directions north, east, south, and west and intercardinal directions northeast, southeast, southwest, and northwest, i.e., for one of eight nominal categories. For this task, the two raters could use any aid necessary, such as their own local knowledge of the area, pictures, or maps, as well as the orientation of lettering included in the maps. Maps were not rated for geometric quality, completeness, or any other parameter. The raters worked independently from each other and both raters rated all sketch maps. Cohen’s \(\kappa \) (Cohen, 1960) was used to assess inter-rater reliability.

Directional ratings were analyzed in two ways, first as categorical data using \(\chi ^2\) tests, and, in a follow-up analysis, with the V-test of circular statistics. Circular statistics transforms angular data (\(\alpha _i\)) into unit vectors of the form \(\vec {x}_i = (\cos \alpha _i, \sin \alpha _i)\); see for example Batschelet (1981) and Berens (2009). The cardinal directions N, E, S, W occurring in our data would thus translate to the vectors (in mathematical angle convention) \(N = (0,1)\), \(E = (1,0)\), \(S=(0,-1)\), and \(W=(-1,0)\). The average of a set of unit vectors, also called the resultant vector, is given by \(\vec {r} = (\sum _{i=1}^n \vec {x}_i)/n\); its length \(||\vec {r}||\) is close to zero if the individual \(\vec {x}_i\) point into different directions and approaches one if the sample is uniform. For a given “theoretical” direction \(\vec {x}_o = (\cos \alpha _o, \sin \alpha _o)\) the V-test tests the hypothesis that the sample is drawn from a population not concentrated in the theoretical direction, i.e., that data cluster around another direction or do not cluster at all. It rejects the null hypothesis if the data are clustered around the theoretical direction.

Before the main experiment started, we performed a pre-experiment to test whether the subjects had a good knowledge of the Tübingen city center and were familiar with the locations that we used in our experiments. Subjects were shown three videos exploring the sketching locations S1, S2, S3. For this, the video goggles were used in open loop. Each video started at a small distance from the sketching location and showed a walk through the immediate environment including views into streets connecting to the target locations. However, the target locations themselves were not visible in any of the three videos. After watching the video, the subject answered the questions: “Do you know this place? Do you know what this place is called? When was the last time you visited this area? Why did you visit this area?”

Three subjects who failed to name all three sketching locations correctly or reported to have last visited them more than 30 days ago were excluded from the main experiment.

After successfully passing the pre-experiment, the experimenter explained the experiment to the subject by reading a note about the procedure of the experiment. The subjects received this note also in writing. The experimenter pointed out the equipment including the rotating chair, the camera, and the fixed square paper. They clarified that the subjects were allowed to rotate on the chair during the experiment whenever necessary and that they would be asked to draw sketch maps of the Market Square (“Marktplatz”, target T1) and the Timber market (target T2) on the paper while still wearing the goggles. Subjects were also informed about a relaxation room which they might want to use should they feel sick during the virtual reality experiment.

At the beginning of each sketching task, subjects were teleported to the respective sketching location facing north. They would then explore the environment by looking around (closed loop VR simulation for rotations only) until they indicated that they had made themselves familiar with the location. Exploration duration was not recorded. The experimenter then asked the participants to imagine the target location and produce a sketch on the virtual sketching board. To do this, they needed to turn into the direction in which the virtual sketching board had been provided. Beyond the sketching board, they would still see the scenery of the current sketching location, facing either towards or away from the target.

Each subject completed two sketching tasks, one for each target location. They were randomly assigned to one of four groups of 20 participants, called A-towards, A-away, B-towards, and B-away. The tasks for each group are listed in Table 1. After finishing the first sketching task, the second was started without delay.

To understand the effect of immersion and spatial presence on the representational memory, we conducted a second experiment in which sketching was repeated after a delay period and outside the virtual environment. We have tested 20 participants (sex not recorded) who were assigned to two groups C-towards and C-away with 10 subjects per group.

Experiment 2 consisted of two phases: Phase 1 for groups C-towards and C-away is identical to experiment 1 of groups A-towards and A-away, respectively, i.e., subjects perform sketching tasks S2-T1 and S3-T2. After the subjects have finished sketching and the experimenter collected the sketch papers, the subjects put away the video goggles and took a 10 min break during which they are were asked to stay in the experiment room. They were offered refreshments or might have a look at books available in the room which were irrelevant to the study. However, they were not allowed to browse in the internet or talk on the phone. After the 10 min break, phase 2 started in which the subjects were asked to draw the layouts of the same target locations as in phase 1 again, but this time without being immersed in the virtual environment nor with any additional information from the experimenter. Finishing the drawing of the target locations concluded the second phase of the experiment and the experiment itself.

In total, each subject produced four sketch maps: two maps of locations T1 and T2 before the break and within the virtual reality environment, and again two maps of the two target locations after the break.

Figure 4 shows four examples of the produced sketch maps together with their orientational ratings. Inter-rater agreement was 100 %, resulting in \(\kappa = 1\) (Cohen’s \(\kappa \), see Methods section). All map orientations are listed in the Appendix. Figure 5 shows the distribution of the orientational ratings for sketch maps produced at the three sketching locations S1, S2, and S3, both for the towards and away conditions. For the statistical analyses, the orientation ratings were binned into four classes north, east, south, and west. The rare intercardinal ratings were counted as 0.5 for each of the two adjacent cardinal directions.

Sketch map orientation in the towards condition for both target locations depends on the sketching location (position-dependent recall). This was revealed by two separate \(\chi ^2\) tests comparing the orientation histograms for the two sketching locations used with each target location. Note that for each target location, the comparison is between different participant groups, as shown in Table 1. The effect is highly significant for both target locations as can be seen in the first row of Table 2. The effect is absent in the away condition.

Sketch map orientations also depend on body orientation. This is apparent from comparing the towards and away condition for individual tasks (pairs of sketching and target location). Again, all \(\chi ^2\)-tests are between different participant groups as specified in Table 1. Test statistics for the effect of body orientation appears in Table 3.

In summary, sketch map orientation is depended (at least) on two factors: sketching location and body orientation.

A further analysis of the data concerns the preferred orientations of the produced sketch maps in relation to three theoretical angles, (i) the airline direction from the sketching location to the target, (ii) the route direction heading into the street that most quickly leads from the sketching location to the target (green and magenta arrows in Fig. 3), and (iii) the canonical view of the target as indicated by the white arrows in Fig. 3. Note that the route direction (ii) is also the body orientation of the subjects in the towards condition. Figure 6 shows the differences of the sketch map orientation and each of the three theoretical angles for all sketching tasks. Thus, if subjects would exactly align their maps with the airline direction, Fig. 6a would show an ideal peak at angular difference 0. Significant alignment with the theoretical angles was tested with the circular V-test (e.g., Berens, 2009). Results show significant effects of all three theoretical angles in the towards condition (airline: \(V(80)=64.4, p<.001\); route: \(V(80)=60.8, p<.001\), canonical: \(V(80)=27.5, p<.001\)), while in the away condition, significant map alignment was found only with the canonical viewing direction (\(V(80)=24.0, p<.001\)). No distinction can be made between the airline and route direction, due to the close agreement of these two theoretical angles in the used sketching and target locations. Furthermore, all theoretical angles are virtually the same (i.e., west) for the S2-T1 task as can be seen from Fig. 3.

In summary, the results of Experiment 1 show that sketch map orientation depends on sketching location in the towards, but not in the away condition. The preferred sketch map orientation in the towards condition is a mixture between the airline or route directions to the target location and a canonical view of the target location, while the sketch maps produced in the away condition show some alignment with the canonical view, but not with the airline or route directions.

Results of experiment 2 appear in Fig. 7. Inter-rater agreement was 100 %, resulting in \(\kappa = 1\) (Cohen’s \(\kappa \), see Methods section). All map orientations are listed in the Appendix. Participants from the C-towards and C-away groups produced sketches of tasks S2T1 and S3T2 before and after the break, i.e., within and outside the virtual environment. In the “towards” and “away” groups, the number of sketch pairs produced with different orientations before and after the break was 12/20 and 8/20, respectively (see data table for experiment 2 in the Appendix). We therefore conjectured that subjects do not simply reproduce their first sketch in the second drawing and tested this hypothesis with two approaches.

First, we analyzed the orientation changes of sketch maps produced by each subject before and after the break using Hotelling’s \(T^2\)-test. For this, the angular deviations between sketches produced before and after the break were transformed into unit vectors with components given by the cosine and sine of the angular differences. This generated a set of bivariate data which were compared to the value \((\cos 0, \sin 0) = (1,0)\) expected under the null hypothesis of equal map orientation. The test did not reach significance for any of the tasks or body orientation conditions (S2T1 towards: \(T^2(2,8)=4.47\), \(p=0.20\); S2T1-away: \(T^2(2,8) = 4.10\), \(p=0.22\); S3T2-towards: \(T^2(2,8) = 5.11\), \(p= 0.17\); S3T2-away: \(T^2(2,8) = 2.70\), \(p=0.35\)).

Second, we conjectured that if subjects would simply reproduce their sketches drawn during immersion also after the break, the effect of body orientation should be identical in the “before” and “after” conditions. In this approach, comparisons are made between the two subject groups C-towards and C-away, and are therefore again be analyzed by \(\chi ^2\) tests. Data were collapsed for sketching task but kept separate for immersion. The “north” bin in task S2T1 was empty for both body orientations in the immersed condition and was therefore deleted. A significant effect of body orientation was found for the sketch maps produced during immersion (before the break, \(\chi ^2(6,40) = 17.5\), \(p =.0051\), \(V= 0.66\)), but not after the break (\(\chi ^2(7,40) = 3.13\), n.s.). Indeed, after the break, when subjects are tested in the lab environment, the definition of the “towards” and “away” body orientations is no longer apparent.

Figure 8 shows the alignment of the chosen view orientations with the theoretical directions airline to goal, route to goal, and canonical view, each with and without immersion in the virtual environment. While immersed in the virtual environment, highly significant alignment was found with all three theoretical directions (airline: \(V(40)=30.8, p<.001\); route: \(V(40) = 24.4, p<.001\); canonical: \(V(40)=20.0, p<.001\)). This pattern is unchanged after the break, when subjects repeat their tasks in an office environment, although the significances are weaker (airline: \(V(40) =9.6, p=.016\); route: \(V(40)=11.9, p=.004\); canonical: \(V(40)=12.0, p=.004\)).

To sum up, the results of Expt. 2 indicate that the imaginary view point assumed during immersion in the virtual environment partially persists when the participants return to the physical environment of the lab.