Instruments such as clarinet and saxophone require a single reed to produce sound. Despite numerous perceptual and physical studies about cane reeds, it is still difficult to relate perceptual qualities to physical parameters. However, the perceptual experiments are often carried out with few subjects that do not repeat the task over time, which calls the reliability of the assessments into question. Some recent results, obtained with 10 experienced musicians that assessed 4 times a panel of 20 reeds, showed that consistent assessments were obtained only at the fourth repetition and for half of the subjects. An explanation may rely in the experimental conditions as all musicians had to carry out a standard procedure: same reeds, same mouthpiece, same global quality assessment task. The current study aims at evaluating the repeatability of perceptual assessments by placing the subjects in natural conditions. As he would normally do when opening a box of new reeds, the musician has to evaluate the quality by using his own material and by defining his own assessment task. The robustness of the perceptual judgments is assessed by repeating the operation 10 times.