Over the past few decades, phycotoxins, secondary metabolites produced by toxic phytoplankton, have seen an increase in their frequency, concentrations, and geographic distribution. As shellfish accumulate phycotoxins making them unfit for human consumption, they are considered as an important food safety issue. Thus, a consumer exposure assessment on phycotoxins is necessary. Exposure assessment requires two types of information: contamination and consumption data. Shellfish contamination data on major toxins encountered by at-risk populations (Domoic Acid group, Okadaic Acid group, and Saxitoxin group) have been reviewed. Consumption data have been reviewed for both general and potential high-consumer populations. Then, we undertook acute and chronic exposure assessments, combining available French contamination data and our own consumption data. Studies including exposure assessment were then reviewed. Lastly, risk characterization was undertaken. It can be concluded that both acute and chronic exposure to phycotoxins via shellfish consumption is a matter of concern, mainly for high consumers identified in this review (specific populations and shellfish harvesters). However, the results for risk characterization must be improved. There is a need for (i) toxicological data to establish a Tolerable Daily Intake; (ii) an assessment of consumption and contamination data, undertaken at the same time, so as to assess exposure.