Bilevel optimization involves an upper-level and a lower-level decision maker. The lower-level optimization problem is nested within the constraints of the upper-level one. A point is said to be admissible for the bilevel problem if it satisfies all constraints and is optimal for the lower-level decision-maker. Bilevel derivative-free optimization (BL-DFO) algorithms address bilevel optimization problems in which either the upper-level or the lower-level problem is solved using a derivative-free optimization method. In this context, existing BL-DFO benchmarking techniques often do not rigorously validate the admissibility of proposed solutions, and do not adequately account for the computational effort deployed by the upper- and lower-level solvers. This work proposes a benchmarking methodology for BL-DFO algorithms. A post-optimization procedure, named refereeing procedure, is introduced to discard non-admissible points and ensure a fair comparison between the algorithms. The computational effort deployed by upper- and lower-level solvers are also taken into account into the overall computational cost. Numerical experiments illustrate the benchmarking methodology.