Introduction Numerous frameworks and tools have been developed to assist medical schools striving to achieve their social accountability mandate. The purpose of this study was to design an instrument to evaluate medical students’ perceptions of the key competencies of a ‘socially accountable’ healthcare practitioner using widely accepted frameworks which contain clear measurable outcomes.

Methods The instrument was designed in three phases: selection of a competency-based framework, development of items, and validation of the instrument through exploratory factor analysis. Medical students in the 6‑year medical degree program at the University of Cape Town, South Africa were invited to participate in the study. Descriptive and inferential statistical analysis was performed using Stata/SE version 13.1.

Results Of 619 students invited to participate in the study, 484 (78%) responded. The CanMEDS framework was selected for designing the instrument, which comprised 35 statements reflecting five competencies for each CanMEDS role. Exploratory factor analysis of the student responses yielded a 28-item instrument. There was a significant difference in overall Perceptions of Social Accountability Instrument (PSAI) scores between men and women (p = 0.002) but no significant difference between the overall PSAI scores for students in the respective years of study.

Discussion This study describes the design of an instrument to evaluate medical students’ perceptions of the essential competencies of socially accountable healthcare practitioners. Used longitudinally, the data may provide evidence of the successes of our programs and identify areas where further improvements are required.