Segmentation-based anomaly detectors proceeds to the clustering of the hyperspectral image as a first step. However, most of the well-known clustering methods cluster anomalous pixels as a part of the background. This paper presents a new hyperspectral image clustering approach based on the betweenness centrality measure. The proposed approach starts by the construction of an adaptive spatial and spectral neighborhood for each pixel. This neighborhood is based on the selection of the nearest spectral and spatial neighbors in multiple windows around each pixel to allow well-suited representation of the image features. In the next step, this neighborhood is clustered based on the edge betweenness measure algorithm that splits the image into regions sharing similar features. This approach (1) allows the reduction of intercluster relationship, (2) favors intracluster relations, and (3) preserves small clusters that can hold anomalous pixels. Experimental results show that the proposed approach is efficient for clustering and overcomes the state of the art approaches.