- Analytical Approaches for Improving the Quality of Observational Workplace-based Assessments
- Approaches to Continuing Competence in a Multiprofessional Context
- Benefits, Challenges, and Tensions in Evidence-informed Regulation
- Building Transparency in International Medical Education
- A Comprehensive Approach to a Quality Improvement Program in Manitoba, Canada
- Conceptualizing Fairness and Identifying Bias in Assessments of and for Learning
- Connecting the Dots: How Licensing Exams Can Provide Value in Predicting In-practice Outcomes
- Content Analysis of Patients’ Complaints Lodged with the Medical Council of Malawi from 2007 to 2011
- A Continued Competence System in New Zealand
- Developing a National Obstetrics and Gynecology Certification Examination in Ethiopia
- Ensuring Continued Competency beyond Fellowship Training: Experience from Hong Kong
- Ensuring Ongoing Physician Competency with CATALYST
- Evaluating Data Models for Continued Competency Assessment
- Evidence on Risk and Support Factors That Affect Physician Performance: A Practical Self-awareness Application
- From Cultural Competence to Cultural Safety to Health Equity
- Balancing “Cop” vs. “Coach”: How Can We Enhance the Value that Stakeholders See in Regulators?
- How Quality Assurance and Quality Improvement Are Flourishing Together in the UK
- The Impact of Remediation on Practice Enhancement in a Quality Improvement Approach to Physician Assessment
- Identifying Physician Clinical Competency Using Licensing Questions and Licensing Types
- Leveraging Institutional Improvement Activities for Continuing Specialty Certification
- Leveraging Technology to Facilitate Assessment and Learning
- Medical Council of Canada 360: A Multisource Feedback Program for Quality Improvement in Physicians
- Quality Assurance vs. Quality Improvement: What Is the Medical Regulator’s Role?
- The Reflective Practitioner
- Shifting the Performance Curve Using Regulatory Data: A Tale of Two MD Snapshots
- Specialty Certification and the Likelihood of Receiving Disciplinary Actions in the United States
- Supporting Physicians Who Are New or Returning to Clinical Practice: The Irish Experience
- The UK Revalidation Model: Experience and Learning from the First Five Years
- The Use of Patient Feedback in Continued Competency Systems: Experience from the General Medical Council
Analytical Approaches for Improving the Quality of Observational Workplace-based Assessments
Assessment of performance based on observation in the workplace can provide critical evidence about the skills and competencies of practitioners at all levels of training and practice. Despite the value of this type of data, little research has investigated ways of improving the quality of data collection instruments. This session is intended to provide participants with a practical approach to applying basic analytic procedures to improve workplace-based assessment instruments, with a focus on best practices for the development of workplace-based assessments.
The presenters collected observational assessment data on first-year pediatrics residents. Eight of 40 assessment items were initially flagged for review by statistical analysis. Of these items, three were deleted, and five were substantially revised to address issues relating to scale use, wording, differences in item applicability across residency programs, and inability of the item to assess observable behaviors. A second data set was produced using the revised instrument. Statistical analysis of the revised instrument showed substantial improvement in performance at both the item and instrument levels. Generalizability analysis allowed for further evaluation of whether the item revisions led to changes in instrument reliability. The generalizability coefficient increased from 0.69 for the initial data collection to 0.75 for the revised instrument.
The results of this study suggest that using straightforward statistical methods to evaluate the quality of workplace-based observational assessment items and inform item revisions can lead to substantial improvements in item and instrument performance. Collecting data about performance in the authentic clinical environment can provide critical evidence to assess the ability of trainees and practicing physicians to perform the activities that are necessary for safe and effective practice.
This research provides evidence in support of using simple statistical approaches to evaluate item performance. Implementing these approaches can lead to improvements in the observational assessment instruments that are used to collect critical performance data. Perhaps most importantly, these types of practical, straightforward approaches to evaluating the quality of assessment instruments can lead to increased confidence in the outcomes.
- Identify at least two approaches to evaluating the quality of observational assessment instrument items
- Describe how specific patterns of results provide insight into item performance
Approaches to Continuing Competence in a Multiprofessional Context
The session will outline developing approaches to continuing competence across the National Registration and Accreditation Scheme in the context of an overarching strategy for ensuring professional practice in Australia. Starting with the work by the Medical Board of Australia and Dental Board of Australia, the session will outline the extension of work to other professions covered by the National Scheme.
The presenter will discuss the different factors that affect risk and continuing competence in a multiprofessional context and the range of regulatory tools involved, from traditional compliance models to behavioral approaches. The session will explore how work on continuing competence in the medical profession can be translated for a wider range of health professions.
- Gain an overview of the National Registration and Accreditation Scheme’s overarching strategy for ensuring professional practice
- Examine some developing approaches to continuing competence across professions and summarize the different factors and regulatory tools involved
Benefits, Challenges, and Tensions in Evidence-informed Regulation
Striking an appropriate balance between quality assurance (QA) and quality improvement (QI) is a key goal in medical regulation. Medical regulatory authorities (MRAs) aim to strike a balance between QA and QI in their programs, policies, and initiatives, while also navigating several practical considerations, including the need for time and financial efficiencies, acceptability (both public and professional), and legislative authority. While these and other considerations constrain the work of the MRA, they should not drive how the appropriate balance of QA and QI is defined. The definition should instead be informed by evidence and analysis, not mere anecdote. Knowing which QA and QI activities are effective and impactful, as well as refining and continually improving these activities, is at the core of public protection and transparent governance.
In addition to informing MRA activities and policies, the lessons learned from QA and QI of practicing physicians should be viewed as the long-term “practice outcomes” of the educational and remedial processes established by the medical profession. These outcomes are essential to improving medical education at all levels, from entry to practice to retirement. A regulatory approach that focuses primarily on data, rather than values and habit, can more effectively justify and support regulatory programs and decisions.
This workshop will build on previous presentations and publications that began the discussion of the value of research in informing regulatory policies and activities. Workshop facilitators will provide a brief overview of the potential benefits and challenges of evidence-informed regulation. Case studies of successful MRA partnered research activities will be presented, and participants will be given helpful tips about research-informed regulation that will apply regardless of the size, resources, or stage of development of their MRA.
Through small group discussions, participants will tackle a series of questions designed to explore the relevance of evidence-informed regulation to their organizational mandate. The questions will challenge participants to think about the implications of evidence-informed regulation from various perspectives through a program evaluation lens. Topics such as resource requirements, data sharing, confidentiality, and acceptability will be covered. Participants will be challenged to consider program evaluation and research as essential to their public protection and professional regulatory mandates.
- Identify the potential benefits and challenges associated with an evidence-informed regulatory approach
- Understand the potential areas in which evidence-informed activities may assist medical regulators in reaching their organization’s goals and objectives, as well as the challenges that MRAs may face in implementing evidence-informed policy
- Discuss and understand some of the research challenges associated with evidence generation
Building Transparency in International Medical Education
In response to recent Educational Commission for Foreign Medical Graduates (ECFMG) investigations of questionable business practices at a small number of medical schools, several initiatives are aiming to increase transparency in international undergraduate medical education to help medical students and authorities make better-informed decisions. ECFMG Certification is required for international medical graduates to complete graduate medical education in the United States and obtain an unrestricted medical license.
ECFMG regularly reviews medical schools and correspondingly updates the ECFMG Certification eligibility information listed in the World Directory of Medical Schools. On its website, ECFMG has published guidance for students on selecting a medical school, as well as a guide by country to the medical education credentials eligible for ECFMG Certification. ECFMG and its foundation, the Foundation for Advancement of International Medical Education and Research, have launched an initiative to make the school’s accreditation status available in the World Directory, and ECFMG intends to include this information on reports sent to licensing authorities, graduate medical education programs, and hospitals.
ECFMG is also evaluating the publication of certification rates and possibly other indicators of the success of a medical school’s graduates. To further foster transparency and public trust, as of 2023 medical schools will need to be accredited by an accrediting agency recognized by the World Federation for Medical Education in order for the school’s students and graduates to be eligible for ECFMG Certification.
This presentation is for medical regulatory officials and others interested in how the international medical education community can work to promote greater transparency and thereby support quality improvement. Individuals involved in the quality assurance and accreditation of undergraduate medical education are especially encouraged to attend.
- Identify current initiatives and opportunities to promote transparency in international undergraduate medical education
- Describe the impetus and desired outcomes of these initiatives
- Explore how to convey this information to medical schools to ensure that they are aware of these initiatives and their potential impacts
A Comprehensive Approach to a Quality Improvement Program in Manitoba, Canada
The College of Physicians and Surgeons of Manitoba (CPSM) is charged by legislation to develop, establish, and maintain a continuing competency program for members to promote high standards of knowledge and skills to enhance the practice of medicine in Manitoba, Canada. CPSM had previously employed multisource feedback as a stand-alone mechanism to provide evaluation and feedback to physicians, but a more comprehensive and robust quality improvement program was desired. The new Quality Improvement (QI) program, based on the Federation of Medical Regulatory Authorities of Canada Physician Practice Improvement document, was launched in January 2019.
The QI program includes a cycle of activities for physicians: understand your practice, assess your practice, create your learning plan, implement your learning plan, and evaluate the outcomes. These activities form the basis of a continuous quality improvement cycle. The QI program will require participation from all CPSM members and will operate on a seven-year cycle. All members will complete a physician questionnaire that asks them to describe their practice and work, including practice settings and involvement in teaching, administrative, and other activities. They will submit information about their continuing professional development activities. Some members will also undergo off-site chart audits, receive multisource feedback, and/or undergo on-site office visits with chart review and chart-based discussion.
All participants will receive feedback, either written or via facilitated discussion, and will complete an action plan identifying one or more practice improvement activities that they will strive to complete in the subsequent year. All participants will be contacted after one year to assess the outcomes of their action plan. They will be asked about successes and challenges that they encountered. This process does not result in a pass/fail rating but rather is intended to promote further reflection and ongoing practice improvement. The program has received accreditation by the College of Family Physicians of Canada.
The QI program will use information on physician risk and protective factors to assist in the determination of the category of review. The correlation of these factors with outcomes of the reviews will be studied over time and will help to validate the use of these factors. This analysis will assist regulators in focusing their resources on the members most in need of assistance.
To enable future research opportunities, the program includes a mechanism to connect the data with quality-of-care indicator data held by the Manitoba Centre for Health Policy (MCHP). Data from the QI program could be matched anonymously with MCHP data to allow for analysis of correlations of program outcomes with clinical care indicators. This novel type of research would explore the relationship of program outcomes with actual clinical performance data.
- Identify components of the CPSM Quality Improvement program
- Identify potential research opportunities for quality improvement programs
Conceptualizing Fairness and Identifying Bias in Assessments of and for Learning
Fairness is not an inherent property of a test, but instead stems from the assessment’s validity in terms of score use and the intended or unintended consequences of testing. It should therefore be of primary interest to medical regulators. Standardized test scores can be used to ensure that a performance standard is met by, and equally applied to, individuals entering the field, thereby limiting group-based biases that can occur with other types of assessment, such as face-to-face interviews. Determining how best to detect and contextualize group differences in high-stakes credentialing examinations remains a priority, given the potential impact of score-based decisions on individuals’ careers.
This presentation will review methods for identifying group differences at the test and item levels, provide real-world examples, and place results within the larger context of fairness as it applies to regulatory decisions and the potential impact on test takers and groups.
Questions surrounding fairness tend to focus on examination scores and whether scores differ across groups of interest. To investigate this question, numerous methodological approaches are available. These methods include variations on and applications of regression, such as determination of prediction bias and selection bias, and multilevel modeling. In medical licensure, prior studies have detected group differences across demographic and other examinee characteristics (e.g., location of education), leading to the question of whether such differences are due to social differences, selection patterns, or some form of test bias.
One potential contributor to group differences in test scores is individual item bias, which can be explored through methods to detect differential item functioning (DIF). These methods aim to identify whether individuals from different groups with the same ability level have a different probability of answering an item correctly. DIF studies in the 20th century led to significant interest in DIF as a concept, as well as a movement to explore DIF further. Today’s test development methods, however, tend to limit or minimize DIF, and in those cases where it still exists, DIF can be difficult to explain beyond type I error.
The Standards for Educational and Psychological Testing (2014) provide a multifaceted definition of fairness: that all takers have the opportunity to demonstrate their ability on the construct of interest through standardization of administration and scoring procedures, that any test characteristics not related to the construct of interest are addressed, and that scores are interpreted similarly for all takers. The standards also are clear that group differences do not themselves indicate bias but should motivate investigation into the cause of such differences.
- Describe methods for detecting and contextualizing group performance differences
- Define differential item functioning
- Describe approaches to identify and remediate bias in test design, item writing, and use of assessment results
Connecting the Dots: How Licensing Exams Can Provide Value in Predicting In-practice Outcomes
The primary use of any medical licensing exam is to assure the public that a candidate has demonstrated adequate competency in the domains deemed necessary for entry into independent practice. However, interest in assessing whether licensing exams can be used for additional purposes, namely to predict in-practice assessment measures and clinical performance, is increasing.
Previous research suggests a link between exam performance and certain prescription patterns or disciplinary actions. However, some of this research is decades old and warrants replication in light of changes in medical education and regulation. Over the past year, the Medical Council of Canada (MCC) has been collaborating in two separate projects with the College of Physicians and Surgeons of Ontario (CPSO) and the College of Physicians & Surgeons of Alberta (CPSA) to assess whether Canadian medical licensing exams can be useful in predicting both in-practice measures and prescribing behaviors.
MCC is responsible for developing and administering national exams that are used as part of the licensing process for physicians in the 13 provincial or territorial jurisdictions. This two-part exam program is referred to as the Medical Council of Canada Qualifying Examination (MCCQE). CPSO and CPSA are medical regulatory authorities responsible for registering and licensing physicians in their respective provinces. Furthermore, they manage and administer competence assessment programs that all practicing physicians take part in at given points in their career. The project undertaken with CPSO focused on determining whether MCCQE performance (based on first-attempt pass/fail standing for each exam) was predictive of physicians’ overall peer assessment outcome. The work undertaken with CPSA was aimed at assessing the predictive relationship between first-attempt MCCQE pass/fail standing on each of the two exams and patterns of benzodiazepine and opioid prescribing for a sample of physicians in that province.
This presentation will provide an overview of (1) results obtained in these studies with practical significance for participating organizations, (2) the importance of this work in promoting integrated, longitudinal assessment of physician performance, (3) lessons learned in developing these cross-institutional collaborative frameworks, and (4) the benefits of such collaborations. Issues pertaining to research and data exchange protocols, ethics, and organizational culture will be discussed, and examples will show how these issues were successfully addressed.
This presentation will provide concrete examples of how organizations that play a critical role in medical regulation can collaborate and maximize their resources to fulfill the mandate to protect the public and ensure safe and effective patient care.
- Learn how medical regulation can benefit from a multi-institutional research program
- Understand the importance of assessing physician performance in a longitudinal and integrated fashion
- Describe lessons learned in multiorganizational collaboration
Content Analysis of Patients’ Complaints Lodged with the Medical Council of Malawi from 2007 to 2011
Patient complaints can provide valuable information for medical regulators regarding physician and health facility performance. The Medical Council of Malawi analyzed patients’ complaints lodged between 2007 and 2011 to evaluate the content, source, and outcomes of the complaints.
Specifically, the council aimed to determine the nature of the complaints (communication, clinical care, rights, access); the type of facilities and departments where the complaints originated (private, religious, public; medical, maternity, surgical, pediatrics); the distribution of medical practitioners involved in the complaint (physicians, clinical officers, medical assistants, lab technicians, others); the demographic characteristics of patients/guardians who filed complaints (age, sex, occupation, education, guardian/patient status); and the outcomes of complaints (not upheld, outcome pending, deregistered, reprimanded, warning, remedial action taken).
Notable findings include poor clinical care (39%) and unprofessionalism (27%) as the top two complaints reported, with clinical officers (36%) and doctors (28%) receiving the most complaints; the majority of complaints were in OB-GYN (36%) and medicine (20%), while 42% of the health facilities were public (government), and 29% private clinics (n=88). Of the outcome of complaints, 22% were not upheld, 22% of practitioners were warned, and three practitioners were deregistered.
The goal of the session is to demonstrate that patients’ complaints are important, as they provide feedback on how physicians and health facilities are performing. In addition, the session will inform the attendees on appropriate strategies to use in response to patients’ complaints, such as improving curricula on professionalism, communication skills, and medical practice.
- Recognize the value of patients’ complaints as a mechanism for feedback on performance
- Determine appropriate strategies to address patient complaints
- Understand which medical regulatory bodies need to review and strengthen their monitoring and evaluation of health systems, including health training institutions
A Continued Competence System in New Zealand
Continuing professional development (CPD) has been mandatory for physicians in New Zealand since 1995. In recent years, the Medical Council of New Zealand has focused on strategies to strengthen the CPD model and has established a recertification program for registered physicians that will not only ensure they remain up to date but will also improve their competence and practice. The council has shifted the focus of recertification activities to increase the value to the physician’s practice, with the aim of improving patient care.
Physicians who do not hold vocational (specialist) registration are required to meet the requirements for recertification for general registrants. The experience, learnings, and success gained from the in-practice recertification program for general registrants, implemented in 2012, has provided an evidence base for the new approach to recertification for vocationally registered physicians.
After extensive consultation and engagement with physicians and stakeholders, the council has developed a new recertification model that will set accreditation standards based on the Vision and Principles for Recertification established in 2016 and will ensure that recertification activities are based on evidence of greatest value to the physician’s practice.
Recertification providers will strengthen their programs to suit the scope of practice, maintaining the focus on quality and lifelong learning while increasing the emphasis on reflection on practice, peer support, and use of data to improve patient care. The changes include a requirement for programs to incorporate activities to improve physicians’ cultural competence and provision of culturally safe care. Physicians will need to develop and maintain a professional development plan; hold an annual conversation with a peer, colleague, or employer about their practice; and undertake continued professional development activities from three areas: measuring and improving outcomes; reviewing and reflecting on practice; and participating in educational activities.
Key elements of the new model include the requirement for providers to offer Collegial Practice Visits (previously known as Regular Practice Review) as an optional activity that involves a comprehensive peer review of a physician in one’s practice. Providers are able to tailor recertification programs to be appropriate to physicians’ actual work and relevant to their identified learning needs, career aspirations, and professional development opportunities.
This presentation will focus on the changes that the council is implementing for vocationally registered physicians and recertification providers, as well as the evidence that supports the new recertification approach as a key mechanism for ensuring the continued competence of physicians in New Zealand.
- Gain an understanding of the evidence that has informed the strengthened recertification approach in New Zealand
- Understand the extensive process of engagement with physicians and stakeholders that has contributed to acceptance of changes
Developing a National Obstetrics and Gynecology Certification Examination in Ethiopia
A long-term goal in Ethiopia is to build obstetrics and gynecology (OB-GYN ) residency programs to improve sexual and reproductive health care. With the rapid development of 13 residency programs in Ethiopia, the Ethiopian Society of Obstetricians and Gynecologists (ESOG) saw the need for a residency curriculum and evaluation process harmonized across programs, culminating in a national OB-GYN certification examination under the auspices of the Ethiopian Federal Ministry of Health.
Funding to support curriculum development and training was provided through a grant from the Center for International Reproductive Health Training to the Office of Global Women’s Health of the American College of Obstetrics and Gynecology (ACOG). ACOG provided funding for a three-day workshop in February 2018, conducted by staff from the American Board of Medical Specialties® (ABMS®) for ESOG.
Working with an ACOG representative and the ESOG Examination and Certification Committee, three ABMS staff members led a faculty development workshop for approximately 24 OB-GYN residency program faculty from 12 Ethiopian medical schools. Workshop sessions covered the roles of assessment in postgraduate medical education and provided instruction on the following:
- Writing and reviewing well-constructed multiple-choice questions (MCQs) testing application of knowledge suitable for an OB-GYN certification examination
- Developing test specifications, exam blueprints, and item classification schemes for building a bank of MCQs and assembling examinations
- Interpreting test statistics (reliability, precision) and item statistics (item difficulty/discrimination) for MCQ exams and performing a key validation
- Setting defensible pass/fail standards, including methods for selecting standard-setting participants and setting absolute (criterion-referenced) standards
- Designing examinee score reports to provide information about areas of strength and weakness
After the workshop, ABMS staff facilitated on-site item writing/review sessions to begin development of an item bank, resulting in approximately 80 items approved for future use. The workshop provided faculty with a broad understanding of test development requirements, enabling them to train additional faculty.
After the workshop, ESOG faculty developed additional items and incorporated material provided by ACOG. In June 2018, a 180-item test was taken by residents drawn from all programs and four years of training. Analysis of the test characteristics indicated good psychometric qualities. Future plans are to expand the item pool and subdivide it into items appropriate for in-training and certification examinations. Pilot testing will continue for the next several years, with the first administration of a national certification examination anticipated in 2023.
- Identify major considerations in designing a new certification process that is suitable for its purpose
- Describe a practical approach to begin the development of a new certification examination
Ensuring Continued Competency beyond Fellowship Training: Experience from Hong Kong
In Hong Kong, registered physicians and dentists are required to complete at least six years of supervised training to become Fellows of the Hong Kong Academy of Medicine and be eligible for inclusion in the Specialist Register maintained by the Medical Council of Hong Kong or the Dental Council of Hong Kong.
The academy consists of 15 specialty colleges. It has the statutory authority to organize, monitor, assess, and accredit all medical and dental specialist training, as well as the provision of continuing medical education (CME), which is an ongoing requirement for maintaining specialist registration. The academy’s CME has further evolved in the past decade to encompass continuous professional development (CPD) activities, which take a learner-focused approach to encourage active learning.
Fellows are required to fulfill a minimum of 90 CME/CPD points in a three-year cycle or face termination of fellowship and subsequent removal from the Specialist Register. If a fellow attains two-thirds of the required points in a cycle as a result of certain reasonable causes (e.g., a medical condition), the fellow is allowed to engage in a remedial program to make up for the deficiency. Sixteen categories of activities can be accredited for CME/CPD points, with each point being equivalent to one hour of participation in an accredited activity. Fellows must meet a minimum requirement of 15 “active” CME/CPD points, gained from active participation in quality assurance activities, medical audits, mortality and morbidity meetings, or programs involving improvement of patient care. Five years ago, the academy introduced an electronic system that is supported by a mobile phone app to streamline and automate relevant processes.
Medical credentialing has been one of the academy’s objectives since its establishment in 1993. The academy aims to set standards for medical and dental specialists to maintain and improve the safety and quality of health care services. Each of the academy’s colleges has a credentialing committee that identifies existing and emerging high-risk procedures that require the development of guidelines for credentialing based on established principles and mechanisms. These include criteria for credentialing, ongoing requirements for training and performance, revalidation mechanisms, periodic review of criteria, and approval/appeal mechanisms. Credentialing is thus an evolving and continual process driven by the risks and needs of the community, as well as by the competency of individual practitioners.
- Understand the experiences of the Hong Kong Academy of Medicine in the development and implementation of its CME/CPD system and recent development of mechanisms for medical credentialing
- Recognize the importance of CME/CPD, which aims to maintain and continually enhance the knowledge, skills, and professional competencies of medical practitioners to achieve continuous quality assurance
- Gain insights and reflect on how standards for specialists in Hong Kong are maintained and continually enhanced through the CME/CPD system and the development of medical credentialing driven by the Hong Kong Academy of Medicine
Ensuring Ongoing Physician Competency with CATALYST
The purpose of continuing certification in the United States is to ensure ongoing physician competency in order to safeguard patient safety. In recent years, the traditional high-stakes, multiple-choice examination has been criticized as a cost-prohibitive process that is not relevant to physicians’ clinical practice. In response, some specialty boards, among them the American Board of Anesthesiology, the American Board of Pediatrics, and the American Board of Internal Medicine, have implemented alternative assessment formats that focus on facilitating physicians’ continued learning.
In keeping with its mission, the National Board of Osteopathic Medical Examiners (NBOME) has developed CATALYST TM, a longitudinal assessment designed to provide specialty boards with a potential means of assessing ongoing physician competency. CATALYST is based on findings from cognitive learning that emphasize the value of retrieving previously learned content, providing immediate feedback, spacing questions over time, and interleaving topics.
In conjunction with the American Osteopathic Association, NBOME conducted 16-week pilot studies to gather data concerning how diplomates from three osteopathic specialty boards viewed the CATALYST assessment platform and the assessment process. Participants were recruited from the American Osteopathic Board of Internal Medicine, the American Osteopathic Board of Pediatrics, and the American Osteopathic Board of Obstetrics and Gynecology. Results indicated overwhelming support for the CATALYST platform: of the 196 diplomates surveyed, 95% agreed or strongly agreed that CATALYST would help them stay current in their specialty, and 91% thought it would help them take better care of their patients. Over 98% stated that they would rather answer a fixed number of CATALYST questions periodically than take the traditional recertification examination.
The presenter will describe the use of CATALYST as an assessment format and summarize the outcomes of the pilot studies. The next steps for CATALYST, including the development of a new technology platform, will be discussed. Discussion of lessons learned will assist attendees in exploring or considering potential enhancement of similar programs in their jurisdictions.
- Explain the elements of cognitive learning theory that support the use of CATALYST as a longitudinal assessment
- Describe the outcomes of the pilot studies conducted with diplomates of three osteopathic specialty boards
- Describe the next steps for CATALYST
Evaluating Data Models for Continued Competency Assessment
As knowledge testing to assess an individual’s mastery of content at a single point in time becomes less popular as a way to evaluate continued competency after initial licensure and/or certification, regulators must consider alternative approaches and the potential inferences and comparability of the various longitudinal models adopted by medical specialty boards. Outside of the knowledge testing model, what inferences do the data support about test takers, and what do the scores mean to regulatory bodies?
This session will describe several continuing competency assessment models, including the administration of shorter, more frequent assessments (for example, three assessments in six years), a series of question-a-day models (with or without resources or review of incorrect responses), and assessment-for-learning models (for example, journal articles with associated questions).
Each of these approaches provides information about the candidate’s knowledge, practice, and professional behaviors beyond that offered by a traditional 10-year point-in-time assessment. Unlike a single standardized test result, these models can provide data about the candidate’s learning or ability to remediate knowledge gaps. In addition, these models encourage frequent engagement with the learning and assessment process and can nimbly address important medical topics, such as obesity or opioid abuse, as they become the focus of national attention. When coupled with case review or a knowledge check-in, the assessment can provide multifaceted data about an individual’s knowledge and clinical reasoning, demonstrating whether the individual meets the standard to continue in practice. A question-a-day model can provide data about an individual’s ability to read and respond to questions, learn over time, or seek resources — skills that may be integral to the practice of medicine but are not reflected in the data from a single point-in-time assessment.
The varying approaches to assessing continued competency show that physicians value convenience, clinical relevance, and cost effectiveness. An ideal model would integrate certification activities into clinical practice and reflect a clinician’s performance and knowledge without requiring additional investment of time or money. This ideal does not yet exist, although several integrated approaches have a lower cost and require limited time away from practice. These models do not necessarily provide meaningful data about a practitioner’s knowledge or clinical performance.
The presenters will discuss the strengths and weaknesses of the various point-in-time and integrated approaches that can inform the decision to renew an individual’s credential for specialized practice and some of the benefits (to the test taker, to the regulatory authority, to the public) associated with each.
- Develop familiarity with the models used to assess continued competency of medical professionals, the strengths and weaknesses of these models, and the data gathered from them
- Conceptualize assessment results as answering a question about the participants’ skills or knowledge, and understand how various assessment types complement one another to support a decision about a clinician’s continued competency
Evidence on Risk and Support Factors That Affect Physician Performance: A Practical Self-awareness Application
The aim of this presentation is to demonstrate how evidence on risk and support factors that affect performance can be used to help practicing physicians proactively mitigate and manage risks in practice.
In an interactive, e-learning environment, physicians will be presented with evidence on factors that affect performance, mitigating strategies to decrease risk, resources to assist in quality improvement, and a personalized report. The demonstration represents how empirical evidence can be used practically to develop educational tools for physicians in practice. The demonstration also highlights how e-learning technology may be used to enhance learning for practicing physicians.
- Understand the evidence base regarding the risk and support factors that affect physician performance
- Increase awareness of how regulators can use risk and support factors to enhance physician self-awareness of their own practice
- Gain confidence in practical applications by observing the demonstration of a technology-enabled tool used by the Ontario medical regulator to promote safe medical practices
From Cultural Competence to Cultural Safety to Health Equity
Cultural competence in health care has been defined as having an awareness of cultural diversity and possessing the skills, attitude, and knowledge to function effectively and respectfully toward patients from different cultural backgrounds. However, cultural competence of medical practitioners and health organizations (including specialist colleges) is not sufficient to ensure the delivery of culturally safe patient care and culturally safe training. More recently, cultural competence has been criticized as being practitioner focused, rather than patient centered.
The Medical Council of New Zealand (MCNZ) has established a partnership with indigenous physicians in New Zealand (through Te Ohu Rata o Aotearoa, a Māori medical practitioners’ association) to develop new cultural safety standards for medical practitioners and training providers. Cultural safety is seen as essential to realize the potential of the medical workforce and to deliver health equity for disadvantaged populations, including the indigenous Māori population.
This presentation will review the concepts of cultural competence and safety, including discussion of the new MCNZ statement and framework on cultural safety. The presentation will also examine ways that individual practitioners, training providers, and health organizations can improve their individual and institutional cultural safety to enable workforce development and deliver health equity.
- Gain an understanding of the meaning of cultural competence and cultural safety
- Learn how to improve cultural safety to enable workforce development and promote health equity
Balancing “Cop” vs. “Coach”: How Can We Enhance the Value that Stakeholders See in Regulators?
As a regulator, how do you evolve educational engagement to address healthcare workforce improvement (e.g., performance management and quality improvement) while nurturing professional development, elevating joy in practice, and fostering interprofessional learning that improves care? Roll up your sleeves for an hour-long active learning workshop to examine strategies for balancing “cop” versus “coach” approaches of your regulatory strategies and tactics.
Facilitated by ACCME’s senior leadership and educators from ACCME-accredited Chicago-based organizations, participants will work together to map a strategic framework for using collaboration and regulatory harmonization to increase value and lower regulatory burden. The session will include facilitated collaboration and dialogue, where individuals will participate in TRIZ-inspired problem-solving to address the question of evolving the value of regulators as practice-improvement leaders. Educational leaders from an ACCME-accredited organization will contextualize the group’s discussion by sharing examples from their own experience.
- Collaborate to identify those practices that maximize value for regulatory stakeholders, including clinicians, healthcare systems, the public, and other stakeholders.
- Work with colleagues to ideate simplified approaches to engage physicians and other health professionals in performance and quality improvement through harmonized requirements, interprofessional collaboration, and technological innovation.
- Articulate requirements and values that are shared across educational (i.e. CME), specialty, and regulatory stakeholders with respect to healthcare workforce development and payors.
How Quality Assurance and Quality Improvement Are Flourishing Together in the UK
Seeing quality assurance and quality improvement as being in need of balance misses the point. Quality improvements in clinical practice build on assurance about continued competence. The most cost-effective and well-received quality assurance programs will include the ability for clinicians to plan and prioritize improvement.
In the UK, General Medical Council revalidation processes involve every doctor annually in medical appraisal. The demonstration of continued competence by a portfolio route, underpinned by robust clinical governance, allows the regulator to emphasize both quality improvement and quality assurance.
The presenter will discuss research that revealed some key factors in facilitating peak appraisal experiences that were transformative in terms of improvement. The potential negative impacts of a regulatory burden are well documented. In a climate of increasing pressure on health care provision, the research demonstrated positive impacts of appraisal on patient care and enabled the sharing of good practice. Appraisees were empowered to reclaim their professionalism through reflective practice and the opportunity to define their priorities for quality improvement. Having their work commended provided a morale boost and an incentive to excel. The program offered perceived benefits in terms of retention and reduced burnout.
The workshop format of this session will allow attendees to share examples of good practice that will bring together their global experiences. It will look at how regulatory processes can be built, not balanced, to provide assurance in a way that contributes effectively to quality improvements in patient care.
- Understand the value that the American Board of Medical Specialties®/Accreditation Council for Graduate Medical Education core competencies bring to the international accreditation and certification landscape
- Improve awareness of the role of both specialty-specific and harmonized milestones in the evaluation and comparative analysis of international accreditation and certification programs
- Share insights and lessons learned from the continuous development, improvement, and evaluation of international certification and accreditation programs
The Impact of Remediation on Practice Enhancement in a Quality Improvement Approach to Physician Assessment
In every aspect of health care, the emphasis is shifting from quality assurance to quality improvement. The College of Physicians and Surgeons of British Columbia, in its Physician Practice Enhancement Program (PPEP), has been using a quality improvement approach to quality assurance over the past four years. The program has emphasized shifting the curve of performance and improving quality through peer practice assessment, feedback, coaching, and collaborative work with registrants to encourage self-reflection and learning and to direct registrants’ learning where required.
Previously reported analysis of this approach has shown that physicians believe that the process is worthwhile, that their practice has improved, and that patient care has improved. This presentation will look at how performance on the peer practice assessment and remediation activities (self-directed activities versus directed improvement activities) affects registrants’ perception of the program, leads registrants to make changes to their practice, and improves patient care. The study distinguishes between higher performers and physicians that require follow-up activities or interventions and examines how each group responds to questions on anonymous questionnaires three months after their final assessment report. The registrants are asked to provide their level of agreement with the following statements on a five-point scale: (1) “Overall undergoing a PPEP assessment was a worthwhile experience,” (2) “Overall my practice has changed as a result of undergoing a PPEP assessment,” (3) “My clinical care has improved as a result of undergoing a PPEP assessment,” (4) “My record keeping has improved as a result of undergoing a PPEP assessment,” (5) “My practice management has improved as a result of undergoing a PPEP assessment,” and (6) “My patients receive better care as a result of changes that I have made after my assessment.” Although both groups responded positively to these questions, those in the group requiring follow-up responded more positively.
- Demonstrate that a quality improvement approach to quality assurance leads to practice change and improvement
- Examine data showing that physicians who undertake some form of remediation show greater practice improvement
Identifying Physician Clinical Competency Using Licensing Questions and Licensing Types
Recently, the Maine Board of Licensure in Medicine enacted a new rule requiring that physicians who have not been in clinical practice during the 24 months prior to application demonstrate current clinical competency in order to obtain or renew a full license. The board implemented this new requirement by creating new licensing questions and new license types to identify and differentiate between physicians who currently practice clinical medicine and those who do not and may not have practiced for many years. Previously, the board had renewed physicians’ full medical licenses without inquiring about current or recent clinical practice. The introduction of additional licensing questions and the creation of two new license types — an emeritus license and a revised administrative license — has enabled the board to identify physicians who lack current clinical competency for practice and to issue them nonclinical licenses. After implementation of the rule, the number of nonclinical licenses issued by the board increased from 7 in 2017 to 176 in 2018. The board expects this number to double in 2019. Issuing nonclinical licenses prevents physicians with extended lapses in clinical practice from practicing clinically, which supports the board’s mission to protect the public.
- Learn about alternative nonclinical license types
- Recognize the obstacles encountered while creating new nonclinical license types
- Understand the need for building support within the licensee community when creating new nonclinical license types
Leveraging Institutional Improvement Activities for Continuing Specialty Certification
Specialty board certification in the United States has moved to a continuing certification process, involving participation in practice-based improvement activities as well as assessment, professionalism, and specialty-specific continuing educational activities. The American Board of Medical Specialties (ABMS) Multi-Specialty Portfolio Program™ began in 2009 as a partnership between the Mayo Clinic and three ABMS Member Boards. It has grown to include 20 ABMS Member Boards and nearly 100 organizations. Physicians who meaningfully and actively engage in these organizations’ relevant quality, safety, and performance improvement activities can receive ABMS Maintenance of Certification® (MOC®) credit.
Mellie Villahermosa Pouwels, Interim Program Director of the ABMS Portfolio Program, will describe the history, evolution, and current state of the Portfolio Program, including the types of participating organizations and MOC-approved improvement activities. She will emphasize the potential for leveraging organizational quality, safety, and performance improvement activities for physician continuing specialty certification, as well as the potential to support continuing professional development by recognizing meaningful practice-relevant improvement work that physicians undertake in their practices and work settings. She will also outline plans for the ongoing evolution of the Portfolio Program in the form of potential collaborations with interested bodies outside the United States.
- Describe the ABMS Multi-Specialty Portfolio Program
- Explore the potential to align institutional quality, safety, and improvement activities and specialty certification for the benefit of physicians and the organizations in which they work
Leveraging Technology to Facilitate Assessment and Learning
Advances in technology have increased the possibilities for assessing medical knowledge and clinical judgment through more flexible, continuous, and dynamic models that incorporate learning into the assessment process. In 2017 the American Board of Medical Specialties® developed and launched an online assessment platform called CertLink®. Seven Member Boards are currently piloting CertLink, and early results have been very promising.
CertLink is an innovative technology platform for the creation of online assessment programs that drive physician professional development and learning. It combines the ease and automation of online assessment with the timeliness of immediate scoring, critique, and suggested resources for self-study. CertLink enables physicians to integrate the assessment, learning, and improvement process into their daily practice workflow. This integration helps physicians improve quality, create more efficient practices, and become better-informed practitioners. Innovative technologies like CertLink also help medical regulatory authorities leverage physician performance and behavioral data to better inform continued competency decisions.
The session will include a live demonstration of the user interface to showcase customization options available to fit diverse competency-based assessment models. Participants will hear about early results from participating Member Boards, lessons learned, new functionality in development, and product white-labeling opportunities.
- Demonstrate the CertLink platform and customization options available to fit diverse assessment delivery methods
- Understand the advantages of online assessment and potential ways to integrate it within a continued competency system
Medical Council of Canada 360: A Multisource Feedback Program for Quality Improvement in Physicians
The Medical Council of Canada (MCC) is a national organization that assesses medical students and graduates to provide one of the qualifications required for entry into practice in Canada. The College of Physicians & Surgeons of Alberta (CPSA) is the medical regulatory authority for the approximately 11,000 physicians in Alberta, Canada. As the medical profession moves toward a competency-based approach to continuous practice quality improvement (QI), medical regulators and other stakeholders rely increasingly on a system of continuous and comprehensive assessments and feedback.
Recently, MCC has collaborated with organizations across Canada (including CPSA) to develop a national multisource feedback program for practicing physicians. MCC 360 is a novel multisource feedback tool that elicits both quantitative (numerical) and qualitative (narrative) feedback from physician colleagues, nonphysician coworkers, and patients. These data are supplemented by a self-assessment completed by the participating physician, and the feedback is shared with the physician by a peer facilitator. This facilitation leads to an action plan for QI in the physician’s practice.
MCC 360 is intended to improve the quality of feedback data gathered and to support the interpretation, acceptance, and use of data to support practice QI and patient care. The questionnaire statements used in MCC 360 align with the Royal College of Physicians and Surgeons of Canada CanMEDS framework and focus on the physician’s roles as communicator, collaborator, and professional, areas that are predictive of patient satisfaction, complaints, disciplinary actions, and lawsuits.
The goal of this session is to present the initial results of an evaluation of MCC 360 and related processes, as piloted with 325 family medicine physicians from CPSA. A pan-Canadian team of researchers from MCC and CPSA, with combined expertise in quantitative, qualitative, and mixed methodological research initiatives, collaborated on this project. This study draws on quantitative and qualitative data to assess the extent to which the MCC 360 questionnaires, reports, and processes meet the criteria of a good assessment. Data for analysis include MCC 360 questionnaires completed by the participant, physician colleague, and nonphysician coworker; patient questionnaires; physician participant responses to a post–MCC 360 survey; physician practice data; physician-generated action plans; facilitator reports; and physician participant responses to a six-month follow-up survey. Feedback on MCC 360 was elicited from participating physicians and facilitators in focus groups held in Alberta in March and April 2019.
This session will be relevant to individuals and organizations within and beyond the continuum of medical education, medical regulation, and medical practice.
- Learn about the MCC 360 tool and multisource feedback program currently being piloted with 325 family physicians in Alberta
- Apply the criteria for good assessment to evaluate a multisource feedback tool for physicians
- Explore the initial results from the MCC 360 pilot project and lessons learned thus far
Quality Assurance vs. Quality Improvement: What Is the Medical Regulator’s Role?
As medical regulators strive to be innovative and transparent and to use evidence-based methods, the traditional role of regulatory bodies has expanded beyond quality assurance activities to include quality improvement initiatives and programs for physicians. This evolving role has led to an existential crisis among regulators pertaining to their raison d’être: What is the medical regulator’s role?
The goal of this session is to present how one medical regulatory authority in Canada — the College of Physicians & Surgeons of Alberta (CPSA) — is redefining the role of the medical regulator to incorporate both quality assurance and quality improvement initiatives, processes, and supports for physicians. The CPSA regulates the approximately 11,000 physicians in Alberta, Canada. By law, the CPSA is required to assess and ensure the continued competence and quality of physicians in Alberta on a regular basis to protect and promote the health and safety of the public.
Maintenance of minimal standards or basic competence is mandated in Alberta by the Health Professions Act, and this requirement can be interpreted in the form of quality assurance (QA) initiatives or programs. For example, compliance with the CPSA’s Standards of Practice is a QA initiative, whereas having a process in place to continually elicit feedback from patients or colleagues to improve ongoing physician performance would be considered a quality improvement (QI) initiative.
In 2015, the CPSA developed and introduced a new evidence-based Continuing Competence program, designed not only to assess and ensure physicians’ ongoing competence and performance but also to promote, manage, and support QI and risk management activities customized for each physician and medical practice in Alberta. Based on the understanding of physician performance as delineated by the Cambridge model (competence + individual physician factors + group/system factors = performance), the Continuing Competence program incorporates multiple QA and QI initiatives, including individual practice reviews, group practice reviews for clinics or groups of physicians, quarterly individualized audit-and-feedback data reports on physician prescribing of opioids and benzodiazepines, customized reports to encourage physicians’ self-reflection and mitigation of potential risks to performance, and a novel multisource feedback tool (MCC 360) incorporating facilitated feedback from a self-assessment form and surveys completed by physician colleagues, nonphysician coworkers, and patients.
This session will be relevant to leadership, staff, researchers, and other decision makers in medical and other health professional regulatory bodies. Maintenance of regulated members’ competence is demanded of medical regulatory authorities, but within this quality assurance framework there is room for — and perhaps an obligation to include — quality improvement initiatives for all physicians.
- Learn how one medical regulatory authority in Canada has redesigned its competence program using evidence-based mixed methodologies to incorporate both quality assurance and quality improvement initiatives
- Reflect on your organization’s raison d’être in relation to quality assurance and quality improvement and the potentially evolving role of the organization’s competence program
The Reflective Practitioner
The General Medical Council is the independent regulator for physicians in the UK. The council helps protect patients and improve UK medical education and practice by supporting students, physicians, educators, and health care providers.
Medicine is a lifelong journey that is immensely rich, scientifically complex, and constantly developing. It is characterized by positive, fulfilling experiences and feedback, but it also involves uncertainty and the emotional intensity of supporting colleagues and patients. Reflecting on these experiences is vital to ensure personal well-being and development and to improve the quality of patient care. Experiences, good and bad, provide learning opportunities for the individuals involved and for the wider system.
In September 2018 the Academy of Medical Royal Colleges, the UK Conference of Postgraduate Medical Deans, the General Medical Council, and the Medical Schools Council published a short guide, The Reflective Practitioner, for physicians and medical students. The guidance supports medical students, physicians in training, and physicians engaging in revalidation on how to reflect as part of their practice. The guidance was coproduced by the four organizations because joint production was expected to lead to better reception by the profession. Working in this way had both benefits and challenges but gave the profession a clear voice on expectations around reflection.
Further work from the General Medical Council to support the guidance has included the creation of case studies and supporting materials. The Academy of Medical Royal Colleges and the UK Conference of Postgraduate Medical Deans also jointly produced a tool kit to support the guidance. The tool kit describes the principles for effective reflective practice and includes templates and examples. Following agreement by the chief executives of the UK health care regulators, an interorganizational statement emphasizing the benefits of being a reflective practitioner and endorsing the use of reflection for teams is also being developed. This statement will be published in summer 2019.
This presentation will cover the following topics:
- An introduction to reflection and the 10 key points of being a reflective practitioner
- The background for the guidance, including why it was created and how it was developed
- The benefits and importance of being a reflective practitioner
- Details on the production of the guidance, including the benefits and challenges of collaboration
- Further work with other UK health care regulators on the benefits of being a reflective practitioner, including team reflection
- Understand the benefits of being a reflective practitioner
- Learn how the guidance on being a reflective practitioner was produced
- Explore how the General Medical Council is continuing to work with other health care regulators on the benefits of being a reflective practitioner, including team reflection
Shifting the Performance Curve Using Regulatory Data: A Tale of Two MD Snapshots
The College of Physicians & Surgeons of Alberta (CPSA) is the medical regulatory authority in the province of Alberta, Canada. Along with other regulatory authorities and stakeholders, CPSA has helped to identify, research, and explore factors that may predict or influence physician performance, including competence, individual factors, and system-level factors. These requirements for physicians’ performance can be understood using the Cambridge Model developed by Rethans et al. A physician’s prescribing practices may also be indicative of performance.
In 2016, CPSA conceptualized a series of reports called MD Snapshot, designed as tools for feedback, physician self-reflection, and practice quality improvement. Utilizing the growing body of evidence around factors and CPSA’s own prescribing databases, the MD Snapshot–Practice Checkup and MD Snapshot–Prescribing Profile tools were developed and introduced to physicians in Alberta.
MD Snapshot–Practice Checkup is an annual, personalized report for physicians. Data are compiled from CPSA’s registration and annual license renewal databases, in combination with CPSA’s physician-specific prescribing database. The first iteration of Practice Checkup was sent in November 2017 to all active physicians in Alberta. A survey and focus groups eliciting feedback about Practice Checkup resulted in mixed reactions from physicians, who are acclimatizing to receiving such reports. A revised Practice Checkup was sent in December 2018 to all active physicians in Alberta, and a survey following this iteration stimulated additional feedback about the utility and value of the report.
MD Snapshot–Prescribing Profile is a personalized report for physicians containing numerous prescribing benchmarks. The first iteration of Prescribing Profile was sent in December 2016 to all Alberta physicians who had prescribed an opioid and/or a benzodiazepine to at least one patient in the third quarter of 2016. Additional iterations based on selected prescribing criteria are sent to physicians throughout the year on a quarterly basis. The data are apportioned into two domains: opioids (measured in total oral morphine equivalents prescribed) and benzodiazepines (measured in total defined daily dose prescribed). Two years after the inaugural Prescribing Profile was distributed, a 23% decrease in overall OME prescriptions and an 18% reduction in overall DDD prescriptions were observed. This downward trend has continued over each quarter in which the Prescribing Profile has been distributed.
The goal of this session is to present one organization’s experience with and lessons learned from providing physicians with feedback based on regulatory data, in an attempt to shift the performance curve by promoting self-reflection and quality improvement among all physicians.
- Learn about the development and implementation of two unique tools for physician performance improvement in Alberta, Canada: MD Snapshot–Practice Checkup and MD Snapshot–Prescribing Profile
- Gain an understanding of Alberta physicians’ attitudes, opinions, and measurable performance reactions to these novel tools
Specialty Certification and the Likelihood of Receiving Disciplinary Actions in the United States
In the United States, physicians must, at minimum, graduate from a medical school, participate in a residency program, and pass a licensure examination to be eligible to practice medicine. Many physicians go beyond these minimum requirements. For example, over three-fourths of physicians in the United States become board certified in their specialty. Specialty board certification is intended to indicate a physician’s expertise in a specific area of practice.
In three separate studies, the Federation of State Medical Boards partnered with the American Board of Anesthesiology (ABA), the American Board of Family Medicine (ABFM), and the American Board of Internal Medicine (ABIM) to explore how specialty certification indicates physicians’ expertise, measured in the form of lower risk of receiving disciplinary actions by state medical boards.
The first study, conducted in 2014, involved anesthesiologists who were trained between 1971 and 2011. Compared with anesthesiologists who passed both the written and oral examinations for their primary ABA certification on the first attempt, those who did not pass either examination (hazard ratio = 3.60) and those who passed only the written examination (hazard ratio = 3.51) had an increased risk of receiving a disciplinary action.
In the second study, among 120,443 family physicians, having ever been ABFM certified was associated with a reduced likelihood of receiving a disciplinary action (odds ratio = 0.35). The third study looked at 66,881 physicians who were part of an accredited internal medicine residency program from 1995 to 2004 and found that the rate of having received disciplinary actions was lower for ABIM certified physicians (1.2%) and other specialty board certified physicians (2.4%), compared with their noncertified counterparts (6.0%).
All three studies provide evidence that board certified physicians are less likely to receive disciplinary actions by state medical boards, compared with physicians who are not board certified.
- Learn about the key aspects of medical regulation in the United States
- Describe specialty board certification in the United States and its role in medicine
- Understand the highlights of studies that the Federation of State Medical Boards completed with three specialty certification boards examining the relationship between board certification and disciplinary actions
Supporting Physicians Who Are New or Returning to Clinical Practice: The Irish Experience
Ireland has a high reliance on international medical graduates, with approximately one in three registered physicians obtaining their basic medical qualification outside Ireland. The Medical Council recognizes that entering practice in a new health system for the first time can be challenging. In response, the council developed the Safe Start initiative to support physicians’ safe entrance into clinical practice in Ireland.
The first step in the development of the Safe Start initiative was identifying challenges faced by physicians who are new or returning to practice in Ireland. The Medical Council conducted a three-stage, mixed-methods research study to establish the educational needs of this group of physicians. The research included a literature review, a quantitative survey, and qualitative semistructured interviews with key informants, including international graduates and senior physicians, using structured topic guides.
The consultation identified the following nonclinical practices as key to supporting physicians’ transition to clinical practice. In the cultural, legal, and ethical practice of medicine, key practices are (1) knowing the legal requirements for prescribing in Ireland (including controlled drugs), (2) dealing with end-of-life ethical and legal issues, (3) obtaining consent appropriately where a patient does not have capacity, and (4) dealing with a patient’s medical information. In the area of communication, the following were identified as important: (1) communication skills to support patient-centered practice; (2) communication skills to support challenging discussions, including giving bad news and engaging in end-of-life care discussions; (3) advocacy skills for the transfer and escalation of care; and (4) specialist communication needs, especially for physicians from societies where cultural norms differ.
These findings informed the development of the Safe Start resource, which addresses common clinical practice scenarios that may arise in the areas of consent, prescribing, end-of-life care, medical record keeping, professional conduct and ethics, communication skills, and physicians’ well-being. The second phase of the Safe Start initiative will focus on ensuring that physicians who are new to practice are engaging in relevant and mandatory continuing professional development to facilitate the provision of safe, quality health care.
- Apply knowledge to develop tools within participants’ own jurisdictions
- Examine processes to determine relevant continuing professional development
- Understand how to modify continuing professional development programs to support physicians who have knowledge and experience gaps
The UK Revalidation Model: Experience and Learning from the First Five Years
At the end of 2012, medical revalidation was launched in the UK as a system to provide assurance of continued competency through a focus on continuous improvement, learning, and reflection. Since then, all physicians registered with the General Medical Council (GMC) who wish to hold a license to practice in the UK have had to participate in revalidation. Revalidation is based on an annual appraisal in which physicians reflect on information about their scope of practice and information from local clinical governance systems about their fitness to practice.
To evaluate the impact of revalidation, the GMC funded a three-year independent research study, which was published in May 2018, and commissioned a review of the impact of revalidation. The report of this review, Taking Revalidation Forward, was published in January 2017. Key areas that the research explored were (1) how successful the model has been in ensuring that all physicians are working within a clinical governance system that regularly evaluates their fitness to practice and (2) the impact of revalidation on physicians and health care organizations.
The presenter will share the key findings from the independent reviews and changes that have been made to the model thus far. In particular, the presenter will explore the benefits and challenges of the UK revalidation model for physicians working outside of mainstream practice, such as those with portfolio careers or in nonclinical practice and will discuss how the GMC will continue to refine the model for these physicians.
- Gain insight into the findings of the independent research into the UK revalidation model and hear about the impact of revalidation on physicians and health care organizations
- Understand the benefits and challenges of the UK revalidation model for physicians outside of mainstream practice
- Learn about the changes the GMC has made and intends to make to the revalidation model
The Use of Patient Feedback in Continued Competency Systems: Experience from the General Medical Council
The approach to patient feedback taken by the Genral Medical Council (GMC), as part of the system of revalidation, has developed since its introduction in 2012. This presentation will include a preview of results of a recent public consultation about proposed changes to the patient feedback requirements. Attendees will understand how patient feedback can allow physicians to better understand how patients experience the care they give, as well as the role of patient feedback in identifying areas of success and those in which improvement is needed.
As part of the appraisal for revalidation, physicians reflect on information about their practice. One type of information is feedback from patients. Currently, physicians collect feedback from a sample of patients at least once every five years, using a structured questionnaire. This system was introduced in 2012 in response to the immaturity of patient feedback mechanisms at that time.
The GMC has considered results of two independent reviews of revalidation — Taking Revalidation Forward by Sir Keith Pearson in 2017 and an evaluation by the UK Medical Revalidation Evaluation Collaboration (UMbRELLA) in 2018. The reports showed that, while physicians indicated that patient feedback is the most helpful type of information, the mechanisms used to collect it have shortcomings. The GMC is committed to revising the requirements to address some of these issues and to improve the value of feedback for physicians.
This presentation will cover the following topics: (1) why and how the GMC introduced its original patient feedback requirements for revalidation, (2) experiences and lessons learned since 2012 (including results of independent reviews of revalidation), and (3) how the results of the GMC’s recent public consultation are being used to shape the patient feedback requirements. A summary of findings from the consultation will be provided, with recommendations for changes to improve the process.
- Demonstrate awareness of how and why the GMC implemented its model of patient feedback in 2012 as part of its continued competency systems for physicians
- Describe some of the challenges that physicians and patients can face when all physicians are required to take a single approach to patient feedback as part of a continued competency system
- Explain how the GMC intends to revise its requirements for physicians to reflect on patient feedback in light of the results of a recent public consultation