We present in this paper GM-DVFS, an adaptive DVFS scheme for energy efficient decoding of H.264 videos. GM-DVFS uses metadata (normalized by the MPEG Green Metadata standard) providing information about the upcoming workload. In the solution we propose, these metadata are processed within an adaptive filter to build dynamically an accurate video decoding complexity model. This model serves to calculate the minimal required processor frequency for decoding a video frame while guaranteeing the real time constraints. Our performance evaluations showed that the proposed algorithm is able to converge to an accurate complexity model (4%) in less than 1 second (in the worst case). Moreover, it is simple to implement (250 lines of C code) and induces very low overhead (1400 cycles/frames). On the other hand, it allows to achieve up to 46% energy saving as compared to the on demand Linux DVFS governor.