Grace Thiel OMS III, Troy Puga OMS III, Morgan McBride OMS III, Charles Marchese OMS III, Jessica Gerczynski OMS II, Larry Segars PharmD, DrPH, Jennifer Dennis PhD, MS
Kansas City University

ABSTRACT

Introduction: Orthopedic surgery residency programs emphasize research output, including abstract presentations and publications. Previous research has examined publication trends for allopathic orthopedic surgery residents, but no study has examined the trends for orthopedic surgery residents from previous American Osteopathic Association (AOA) programs. This study assessed research outputs of participants in former AOA orthopedic surgery residency programs in relation to a future in academic medicine. Methods: Former AOA orthopedic surgery residency programs with a graduating class from 2014-2016 were evaluated from 2022-2023. Doximity and AOA residency programs were reviewed to identify graduates. Program coordinators were contacted to determine missing or incomplete information. Publications specific to each graduate were identified using Scopus, PubMed, and Google Scholar from pre-residency, residency, and post-residency timeframes. Descriptive statistics were applied per graduating class and publication timeframe. An independent t-test or Mann-Whitney U was utilized to compare data between select variable groups. A binary logistic regression or linear regression was utilized to evaluate for possible associations between study variables and categorial (fellowship, career type, first authorship) or continuous (h-index, number of first authorships) variables. Results: A total of 135 residents with 363 publications were included. Publications increased over time with a significant correlation between the number of publications during and after residency (p=<0.001). Residents who pursued a fellowship (p=0.02) had a higher h-index score. Discussion: Research productivity is important to orthopedic surgery, particularly outcomes that continue to increase in quantity over a given resident’s career and educational journey. 

Keywords: orthopedic residency programs, research output, graduate medical education

INTRODUCTION
Early in their training, most medical students consider what specialty of interest and how they can make themselves more competitive for residency. Historically, many of the more competitive specialties, including orthopedics^1,2, encourage applicant distinction through various means, such as research publications^3,4. It has been reported that program directors look for a variety of research activities when assessing possible residency candidates^3,4 including poster and oral presentations, and publications. With transitioning to a single match process among allopathic and osteopathic medical schools, the motivation and sense of necessity for osteopathic medical students to distinguish themselves has only increased. This is especially true for students aiming for more competitive specialties, like orthopedics, where an osteopathic education is perceived as a barrier to successfully matching given the recent transition to a single match⁵. 

In 2021, the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX) and United States Medical Licensing Examination (USMLE) board examinations made the transition to a pass/fail system for the first examination. Previously, numerical scores for Level 1 (COMLEX) and Step 1 (USMLE) examinations were often used for selecting students for residency interviews; however, with the pass/fail system residency programs may need to focus more intently on other metrics to select students for interviews^6,7, including research output. Additionally, the AAMC reported that 83% of the surveyed programs utilize a pass/fail grading scale for preclinical years⁸. Overall, the transition to pass/fail systems for both board examinations and curricular grading schema present a challenge as this creates decreased opportunities for students to distinguish themselves with scored metrics, while simultaneously increasing the importance of other metrics such as research output for the residency application process. 

In addition to the challenges perceived by medical students, residency programs are required to incorporate some form of research in order to remain accredited with the Accreditation Council for Graduate Medical Education (ACGME)⁹. Hence, while research outputs in medical school offer an opportunity for students to help distinguish themselves as an applicant, it also suggests that the student has research experience and therefore will need less guidance for future projects. Residents, much like medical students, must make themselves competitive for fellowship programs. Certain fellowships remain very competitive with low match rates, including trauma fellowships within the orthopedic surgery field, which have previously been the most competitive over the past decade. Trauma fellowship directors highly rated research experience and interest in an academic career when selecting an applicant¹⁰. 

Although previous reports^3,4,11,12 have noted the importance of research output as it relates to allopathic residency programs, there is no information specific to previously distinguished AOA programs. We aim to address this gap and the notable exclusion of previous AOA affiliated residency programs. With this study, we aim to look at orthopedic residency programs that were previously designated as AOA programs regarding 1) research outputs per resident during pre-residency, residency, and post-residency timeframes, 2) h-index scores, 3) fellowship participation, 4) future field of practice (private or academic), and 5) sex. 

METHODS
Study design and program selection
The study was cross-sectional in design (IRB# 1973346-1) to assess the peer-reviewed publications for residents of orthopedic surgery residency programs previously designated as AOA. All residency programs listed as transitioning to ACGME accreditation were eligible for inclusion¹³. The American Osteopathic Academy of Orthopedics (AOAO) website was used as a second source to confirm that all programs were appropriately included¹⁴. All programs were confirmed to have a graduating class in 2014, 2015, and 2016. Programs without graduating classes during the study timeframe, or those that had closed and graduate data was unavailable, were excluded. 

The names of graduating residents from 2014-2016 were searched on Doximity and the specific programs’ websites. If program graduates were not listed on these websites, program coordinators were contacted. Program exclusion followed non-response from program coordinators. Google was utilized to find additional information about each resident, including 1) if they pursued a fellowship and what subspecialty it was in, 2) if they went into academic medicine, and 3) the resident’s sex.

A list of publications per resident was compiled through a search on Scopus, followed by Google Scholar and PubMed, serving as a cross-check to confirm all documented publications. Resident-specific information was determined including: 1) the number of first author publications, 2) the number of last author publications, 3) the names of journals published in, and 4) the type of publication (i.e. case report). 

The h-index score was obtained per resident through Scopus as it includes all publications, regardless of authorship order, in its calculation¹⁵. If a score was unavailable on Scopus, a manual calculation of the h-index was performed by an author (GT) using a standardized formula¹⁶.

Data Analysis
Data was collected and publications were sorted into multiple timeframes: pre-residency (including both undergraduate school and medical school), residency (five-year span for US orthopedic surgery programs), and post-residency (through February 24, 2023, and included the fellowship timeframe if one was completed). The timeframes of pre-residency and residency were expanded to six months after completion to account for work completed and submitted during the previous timeframe¹¹. 

Publication number was determined per graduating class and publication timeframe using descriptive statistics. A Pearson product moment correlation coefficient was utilized to assess the relationship between the number of publications for the training timeframes. A Levene’s test suggested the samples had unequal variances, thus a Mann-Whitney U test was utilized to compare publication means and whether or not a future in academic medicine was pursued. An independent sample two-tailed t-test was utilized to compare publication means and fellowship participation. Both two tailed t-tests and Mann-Whitney U tests were also utilized to examine for possible differences between the number of publications before residency completion (pre-residency and residency timeframes) and the fellowship subspecialty. A Mann-Whitney U test was used for the fellowships of Sports Medicine, Shoulder/Elbow, and Foot/Ankle due to violation of the continuous normality assumption. All remaining were examined with an independent sample two-tailed t-test.  

A binary logistic regression was utilized to assess the relationship between career type (academic or private practice), fellowship, and first-author publications before residency while controlling for sex. A linear regression was also utilized to evaluate the relationship between the number of first-author publications prior to residency training and the h-index score in regard to fellowship participation and career path (academic or private practice), while also controlling for sex. Additional differences in research outputs amongst the sexes was assessed using a Fischer’s Exact test. A Mann-Whitney U test was utilized to examine the comparison between sex within the academic population and total publication number. An independent sample two-tailed t-test was used to examine the comparison between sex and publications before residency (pre-residency timeframe). Statistical Package for Social Sciences (SPSS version 23, IBM, Armonk, NY) was utilized for all statistical analyses using an alpha value of 0.05. 

RESULTS
A total of 40 programs were identified as previous AOA programs and seven were excluded due to a lack of graduating classes during the study timeframe. Of the 33 programs, 18 programs were included in the study (Figure 1). There were 135 residents included, all of whom were osteopathic physicians, with a total of 363 publications across all timeframes of training (Table 1). The resident population was overwhelmingly male (124, 91.9%), with only 11 female trainees (8.1%). A majority of the residents participated in additional training (n=108, 80.0%), with all female residents (100%) and a majority of male residents (n=97, 78.2%) pursuing fellowship training. Private practice was the most frequent career trajectory (n=119, 88.1%). Of those entering academic medicine (n=16, 11.9%), 12.5% were female (n=2) and 87.5% were male (n=14). 

In evaluating the publication performance by residency program graduation year, each graduating class had a substantial number of publications, however, the 2015 class had two outlier residents with publications that far exceeded their peers. The distribution of these publications specific to residency timeframe, order of authorship, and publication type is shown in Table 2. The total number of publications for each AOA program specific to during the residency and post-residency timeframes is shown in Table 3. 

To evaluate the relationship between publications and timeframes, a Pearson correlation coefficient analysis was conducted. Correlation of publication numbers during and after residency was found to be statistically significant with a medium positive correlation (r=0.59, p= <0.001); a small, nonsignificant, positive correlation between publication rates within the timeframes of before and during residency (r=0.12, p=0.17), and before and after residency (r=0.03, p=0.72) was also noted. Collectively, these data indicate no significant relationship between research productivity before and during graduate training, however, there is a moderate significant relationship between research productivity during graduate training and after its completion. No significant difference was determined when comparing the number of publications between residents in private practice versus academic medicine (U= 849, p=0.45), as well as when comparing the number of publications of those who pursued fellowship versus did not pursue fellowship (t=1.33, p=0.19). The fellowships of Shoulder/Elbow (p=0.01) and Foot/Ankle (p=0.03) were found to be statistically significant, suggesting that there were different publication rates for these subspecialties when the post-residency timeframe was compared to the publication rate before residency completion. All remaining subspecialties had insignificant differences (Table 4).

To evaluate the relationship between career type (academic vs private practice), fellowship, and sex in relation to first author publications and h-index, a binary linear regression was performed. No relationship existed between publications and these variables, however, when examining the h-index score, its relationship to fellowship was statistically significant (Table 5). Females were not more likely to complete a fellowship compared to males, as demonstrated by Fisher’s Exact test, which resulted in a two-sided exact significance value that was found to be nonsignificant (p=0.12). No significant differences were identified using uni- or multivariant analyses when evaluating the relationship between career type (academic vs private practice), fellowship, and the number of first author pre-residency publications while controlling for sex (Table 5).  Finally, analysis of total publication number and pre-residency publications among the two sexes entering academic medicine were found to be not statistically significant (Table 6).

DISCUSSION 
Publication numbers for orthopedic residents trended upwards with progression through their training and career (Table 1). Notably, there was a correlation between the number of publications during residency as compared to after its completion, suggesting that as one better develops the skills necessary for research, they are more likely to continue to produce future projects. However, it is important to consider that these findings, while consistent with previous studies¹¹, might be due to the impact of two substantial outliers in the data, whom had 40+ publications each. When future practice was considered, the results did not suggest that there was an increase in total publication number per orthopedic resident who entered into academic medicine. This was inconsistent with previous reports¹¹, but could be reflective not only of our small sample, but also one that predominately contained residents with few publications. Expanding our sample to include more residency graduation classes per program would likely provide a broader view of the impact the number of publications may have on career trajectory. Indeed, there is generally an expected difference in the publication number needed to competitively apply for faculty appointments for a career in academic medicine. 

There was no relationship to the number of publications and those who pursued a fellowship, suggesting that publications do not have as great of an impact on fellowship acceptance for an AOA-resident program graduates or AOA-affiliated fellowship programs. Similarly, we did not identify a relationship between participation and publication output for the pre-residency and residency timeframes. Segregation amongst allopathic and osteopathic students still exists for most specialties in terms of fellowships¹⁷, so the relationship in publication output expectations could be associated with this. There was a significant relationship identified between Shoulder/Elbow and Foot/Ankle fellowships and research output; both of which had fewer publications than the overall study population. Taken together, these outcomes are in contrast to data reported for allopathic programs¹¹, as well as other studies focused on plastic surgery and urology residents^18,19. We did not find a significant relationship for Trauma fellowships, which came as a surprise as there is an established realm of competitiveness for this subspeciatly¹⁰. Together these results suggest that certain fellowships may look at different metrics when assessing applicants or that a selection bias was present within our study population.

To determine any similarities in metrics characterizing osteopathic orthopedic residency programs, we compared our study outcomes to those publication outcomes for allopathic programs. Indeed, the likelihood of entering academic practice was not impacted by fellowship completion, the number of first author publications before residency, or sex, consistent with previous reports¹¹. Similarly, h-index score had a significant relationship with fellowship pursuit¹¹, suggesting that there was a smaller h-index score for those who did not complete a fellowship. In contrast, a comparison of fellowship completion, career path, and sex related to first author publication before residency, were found to be nonsignificant, as was the relationship between career type and h-index score¹¹. While these results showcase a mix of similarities and differences amongst orthopedic residents from allopathic and osteopathic residency programs, overall, they suggest that in most regards, these residents are the same. One caveat is that first authorship was not as common for osteopathic as compared to allopathic residents, possibly suggesting that experience with research is sufficient as opposed to leading a project. 

Our study found that female residents were just as likely to complete a fellowship or enter the academic field when compared to males. Interestingly, females did not appear to produce any greater total publication number or more publications before residency when population differences were accounted for, suggesting sex did not directly impact publication numbers. Interestingly, of the top 10 specialties, orthopedic surgery has previously reported the lowest number of female residents at 15.4%, a 50% increase from 2004-2005²⁰. This has been coupled with increasing female first authorship in high impact orthopedic journals over an 11-year period²¹. While our study had only 11 women, 100% completed a fellowship and 18% entered academic medicine, demonstrating distinct and increasing success for osteopathic female physicians. 

This study does not come without limitations. Resident classes were first obtained from Doximity, which generally relies on self-reporting. While an effort was made to cross-check, an oversight or accidental inclusion of a resident in an alternate class could have occurred. Resident publication number inaccuracies due to a surname change (leading to fewer publications), a shared name (leading to an increase in publications) or non-indexed publications (leading to underreporting of publications) are possible. Finally, this study spans specific program years, and did not include all former AOA programs due to exclusion criteria and response rate; future studies should continue to assess these research output metrics over time.

CONCLUSION
While research productivity for osteopathic physicians increased over time, our study findings suggest that the associated weight of research might be less for osteopathic physicians as compared to their allopathic counterparts. With the recent changes to undergraduate medical education in regard to pass/fail board examinations and the single match system, research likely will become an important metric utilized by students to distinguish themselves for residency; therefore, supporting research earlier in a student’s career may provide opportunities to distinguish themselves for residency applications.  

Acknowledgements
The authors would like to thank all of the Residency Program Coordinators who helped with the data collection process.

Figure 1 | Figure 2 | Figure 3 | Figure 4 | Figure 5 | Figure 6 | Figure 7

Required Disclosures and Declaration

Copyright Information: No copyright information added
IRB Approval Information: Yes
Disclosure Information: No known conflicts of interest

REFERENCES

1. Cotter EJ, Polce EM, Williams KL, Spiker AM, Grogan BF, Lang GJ. Current State of Research Gap-Years in Orthopedic Surgery Residency Applicants: Program Directors’ Perspectives. The Iowa orthopaedic journal. Jun 2022;42(1):19-30. 
2. Toci GR, Elsner JA, Bigelow BF, Bryant BR, LaPorte DM. Medical Student Research Productivity: Which Variables are Associated with Matching to a Highly Ranked Orthopaedic Residency Program? Journal of surgical education. Mar-Apr 2021;78(2):512-518. doi:10.1016/j.jsurg.2020.08.024
3. Ngaage LM, Elegbede A, McGlone KL, et al. Integrated Plastic Surgery Match: Trends in Research Productivity of Successful Candidates. Plastic and reconstructive surgery. Jul 2020;146(1):193-201. doi:10.1097/prs.0000000000006928
4. Parker SM, Vona-Davis LC, Mattes MD. Factors Predictive of Publication Among Medical Students Participating in School-Sponsored Research Programs. Cureus. Sep 2021;13(9):e18176. doi:10.7759/cureus.18176
5. Mun F, Scott AR, Cui D, et al. A comparison of orthopaedic surgery and internal medicine perceptions of USMLE Step 1 pass/fail scoring. BMC medical education. May 3 2021;21(1):255. doi:10.1186/s12909-021-02699-4
6. Berk GA, Ho TD, Stack-Pyle TJ, et al. The next step: Replacing step 1 as a metric for residency application. Laryngoscope investigative otolaryngology. Dec 2022;7(6):1756-1761. doi:10.1002/lio2.947
7. Lauer C, Buonpane C, Lane S, Foreman Z, Shabahang M. General Surgery Residency Application Evaluation in a USMLE Step 1 Pass/Fail World: A Retrospective Comparison. The Journal of surgical research. Sep 2021;265:317-322. doi:10.1016/j.jss.2021.03.056
8. Colleges AAoM. Grading Systems Used in Medical School Programs. https://www.aamc.org/data-reports/curriculum-reports/data/grading-systems-used-medical-school-programs
9. Education ACfGM. Common Program Requirements (Residency) 2021. https://www.acgme.org/globalassets/pfassets/programrequirements/cprresidency_2022v3.pdf
10. Shaath MK, Warner SJ, Kellam JF, Achor TS. The Competitive Orthopaedic Trauma Fellowship Applicant: A Program Director’s Perspective. Journal of the American Academy of Orthopaedic Surgeons Global research & reviews. May 6 2021;5(5)doi:10.5435/JAAOSGlobal-D-20-00175
11. Carr M, Anderson JM, Shepard S, et al. An analysis of publication trends of orthopedic surgery residency graduates in relation to academic achievement. Journal of osteopathic medicine. Jan 27 2022;122(4):195-202. doi:10.1515/jom-2021-0196
12. Examination USML. USMLE Step 1 Transition to Pass/Fail Only Score Reporting. https://www.usmle.org/usmle-step-1-transition-passfail-only-score-reporting 
13. Association AO. Former AOA Programs That Have Transitioned to ACGME Accreditation. 2021. https://osteopathic.org/wp-content/uploads/2018/02/single-gme-transitioned-programs-opportunities.pdf
14. Orthopedics AOAo. Resident American Osteopathic Academy of Orthopedics Residency Programs by State. 2023;
15. Castillo M. Measuring academic output: the H-index. AJNR American journal of neuroradiology. May 2010;31(5):783-4. doi:10.3174/ajnr.A1888
16. Waterloo Uo. Calculate Your Academic Footprint: Your H-Index. 2023. https://subjectguides.uwaterloo.ca/calculate-academic-footprint/YourHIndex#:~:text=To%20manually%20calculate%20your%20h,author%27s%20h%2Dindex%20is%206.
17. Program NRM. Results and Data Specialties Matching Service, 2022 Appointment Year. 2022. https://www.nrmp.org/wp-content/uploads/2022/03/2022-SMS-Results-Data-FINAL.pdf 
18. DeLong MR, Hughes DB, Tandon VJ, Choi BD, Zenn MR. Factors influencing fellowship selection, career trajectory, and academic productivity among plastic surgeons. Plastic and reconstructive surgery. Mar 2014;133(3):730-736. doi:10.1097/01.prs.0000438043.98762.51
19. Thompson RH, Lohse CM, Husmann DA, Leibovich BC, Gettman MT. Predictors of Scholarly Productivity, Pursuit of Fellowship, and Academic Practice Among Urology Residents Using Medical Student Application Materials. Urology. Oct 2018;120:49-55. doi:10.1016/j.urology.2018.07.016
20. Van Heest A. Gender Diversity in Orthopedic Surgery: We All Know It’s Lacking, but Why? The Iowa orthopaedic journal. 2020;40(1):1-4. 
21. Hiller KP, Boulos A, Tran MM, Cruz AI, Jr. What Are the Rates and Trends of Women Authors in Three High-impact Orthopaedic Journals from 2006-2017? Clinical orthopaedics and related research. Jul 2020;478(7):1553-1560. doi:10.1097/corr.0000000000001043