Using a task-switching variant of dichotic listening, we examined the ability to intentionally switch auditory attention between two speakers. We specifically focused on possible interactions with stimulus–response compatibility. In each trial, two words, one spoken by a male and another by a female, were presented dichotically via headphones. In one experimental group, two animal names were presented, and the relevant animal had to be judged as smaller or larger than a sheep by pressing a left or right response key. In another group, two number words were presented and had to be judged as smaller or larger than 5. In each trial, a visual cue indicated the gender of the relevant speaker. Performance was worse when the gender of the relevant speaker switched from trial to trial. These switch costs were larger for animal names than for number words, suggesting stronger interference with slower access to semantic categories. Responses were slower if the side of the target stimulus (as defined by the relevant gender) was spatially incompatible with the required response (as defined by the size judgment). This stimulus–response compatibility effect did not differ across stimulus material and did not interact with attentional switch costs. These results indicate that auditory switch costs and stimulus–response compatibility effects are dissociable, referring to target selection and response selection, respectively.