Part of recent reforms of postgraduate medical training in the Netherlands is the introduction of formatively intended knowledge testing or progress testing. We previously evaluated the construct validity and reliability of postgraduate progress testing. However, when assessment is intended to be formative, the acceptability of the test (scores) and the educational impact that is achieved are at least as important in the utility of this assessment format. We developed a questionnaire targeted at both educational supervisors and postgraduate trainees, containing questions on general acceptability, educational impact and acceptability of test content. 90 % of trainees and 84 % of educational supervisors completed the questionnaire. The general acceptability of formatively used progress testing is good; however, the self-reported educational impact is limited. Furthermore, trainees query the validity of test content. Formatively intended progress testing is well accepted; however the impact is limited. We discuss the importance of feedback quality and the effect of grading. Furthermore we start a debate on whether, for a genuine effect on learning, formative assessment should have consequences, either by entwining the assessment with the training programme or by linking the assessment to a summative standard.