While medical research continues to investigate the genetic basis of cancer, and personalised prevention gains momentum, little research has been conducted with the individuals who decline predictive genetic testing for cancer. We recruited individuals who had been offered genetic testing for Lynch syndrome or bi-allelic MUTYH mutations due to their participation in a large, population-based, Australia-wide colorectal cancer study. Thirty-three individuals in mutation-carrying families, unaffected by cancer, who had actively or passively declined testing at one of four decision-making points, took part in a qualitative interview about their decision. Data analysis revealed a typology of ‘decliners’: (1) uninformed about genetic testing; (2) a weak intention to undergo genetic testing; (3) conditionally declining; and (4) unconditionally declining testing. In this population we found substantial barriers to achieving the benefits promised by predictive genetic testing; a lack of knowledge of the availability of genetic testing; a lack of trust in genetic test information; a desire to see a stronger benefit from genetic testing before proceeding; and a sense that there may be more negative than positive outcomes from genetic testing. These discourses must be addressed if medical research on the genetic basis of cancer continues to be funded, and personalised prevention of cancer continues to be recommended by experts.