A large number of smart grid pilots are initiated worldwide to explore the potential of smart grids. At this stage, there is a clear need to identify how the results of these pilots can be evaluated. To enable the use of the methodological approaches applied for the estimation of the costs and benefits of smart grids, the results need to be generalized based on specific input assumptions. To this end, it is required to unravel the input-output relationship between the smart grid input and the change in the consumers’ load profile. Therefore, when evaluating smart grid pilots, it is important to take into account which smart grid input variables are used to stimulate load shifting. These input variables are defined by the smart grid concept, and reflect the objectives of the involved stakeholders. This paper describes the challenges of and requirements for the evaluation of smart grid concepts and pilots. Furthermore, the proposed evaluation method is illustrated by describing two different smart grid concepts and their pilot setup