It has been shown that the capacity of visual working memory (VWM) is a strong predictor of individual intelligence, and researchers have developed various training protocols to improve VWM capacity. However, it seems that whether the fidelity of internal representation in VWM can also be improved by training is largely overlooked in the past literature. Here, we introduced a new training approach that involved increasing the perceptual difficulty of training materials to enhance VWM, and both memory capacity and the fidelity of representation were examined to assess the training efficacy. Participants with normal vision and cognitive abilities received 3-week training on VWM using a change detection task, and the results showed that both the capacity and the fidelity of memory representations were improved for training with perceptually difficult stimuli, while only the fidelity was improved for training with perceptually normal stimuli. In addition, we found that the training effects on memory precision may be subject to capacity constraints. We suggest that long-term adaptive training with perceptually difficult stimuli may facilitate encoding efficiency through familiarizing trainees with an increased baseline of cognitive workload during the encoding process. The present study offers clear evidence that training with high perceptual difficulty is more effective and the improvements in VWM are more stable than training with perceptually normal materials, and the simple manipulation on training stimuli indicates that the method can be generalized to a wider range of training situations and populations.