1. Brandon Klein DO, MBA – Northwell Health – Huntington Hospital Orthopedic Surgery Residency
2. Joshua Giordano DO – Northwell Health – Huntington Hospital Orthopaedic Surgery Residency
3. Soumya Polavarapu OMS(4) – Lake Erie College of Osteopathic Medicine
4. Jeremy Reich OMS(4) – Philadelphia College of Osteopathic Medicine
5. Peter White DO, MS – Northwell Health – Huntington Hospital Orthopedic Surgery Residency
6. Randy Cohn MD – Northwell Health – Huntington Hospital Orthopaedic Surgery Residency, Associate Program Director
7. Kanwarpaul Grewal DO – Grewal Orthopedic and Spine Care

Statements and Declarations
There was no funding required for this study. Authors have no relevant financial, non-financial interests, or other proprietary interests to disclose. 

Abstract

Purpose
The study was designed to provide an updated analysis of the spine section of the Orthopaedic In-Training Examination (OITE).

Methods
A cross-sectional review of OITE spine questions from 2009-2012 and 2017-2020 was performed. Subtopics, taxonomy, references, and use of imaging modalities were recorded and changes between the time periods were analyzed.

Results
Trauma was the most common subtopic across all years, comprising 27.3% of spine questions in 2009-2012 and 20.2% in 2017-2020 (p=0.238). Questions on degenerative disease increased by 8.7% (4.5% to 13.2%) in the later subset (p=0.037). The utilization of Computed Tomography (CT) imaging was more common in recent examinations (p=0.250), but MRI remained the most common imaging modality in both subsets. The most commonly cited journal was Spine (26.0%), followed by the Journal of the American Academy of Orthopaedic Surgeons (11.5%) and the Journal of Bone and Joint Surgery (8.2%). The number of references per question decreased from the early to the late subset (p=0.008), while the lag time of references was lower in more recent examinations (p=0.019).

Conclusion
Recent OITE examinations (2017-2020) had an increase in spine questions relating to degenerative disease, decrease in number of references cited per questions, and decrease in lag time of references, when compared to the earlier examinations (2009-2012). Subtopics, taxonomy, and the utilization of imaging modalities did not show significant change. 

Keywords
OITE, Resident Education, AAOS

---

Introduction
The Orthopaedic In-Training Examination (OITE) was developed by the American Academy of Orthopaedic Surgeons (AAOS) in 1963 to determine if each residency program was maintaining minimum standards for orthopaedic education and achieving educational goals.¹ At present, the examination consists of 275 questions evaluating 11 different domains including basic science, foot and ankle, hand, hip and knee, oncology, pediatrics, shoulder and elbow, spine, sports medicine, trauma, and practice management. The test is currently administered in computer format to over 4,000 residents across 20 countries each year.² 

At the conclusion of residency training, graduating orthopaedic residents take the ABOS (American Board of Orthopaedic Surgery) Board Certification exams. Although several studies have found a correlation between OITE performance and passing the Board Certifying exams, the two tests have historically been written by different organizations without a common blueprint.^3-7 Most recently, a review of OITE and Part I Certifying Exam scores from 2014-2018 found an increasing correlation of performance on the two exams associated with increased residency training, noting a correlation of 0.504 for senior residents (PGY-4 and PGY-5s). However, a minimum OITE score correlated with passing the Part I Certifying Exam was not identified.⁷ The value of the OITE has been questioned due to its lack of alignment with the Part I Certifying Exam. Recently, the ABOS and AAOS have begun to collaborate “to identify the score on the AAOS OITE that approximately corresponds to the minimum passing performance level on the ABOS Part I Certifying Examination” by developing questions that are included on both examinations.⁸ A linking study found that the minimum score on the 2021 OITE that corresponded to passing the ABOS Part I Certifying Exam was 69.2%, but this was noted to be applicable only to the 2021 OITE examination.⁹ In 2021, the ABOS developed a common examination blueprint to further align the OITE with the Part I Certifying examination.⁸ 

Therefore, it has become increasingly important for orthopaedic residency programs to guide resident learning in preparation for the annual OITE examination. To the authors’ knowledge, there has not been a recent analysis of the spine section of the exam. The purpose of this study was to assess recent trends of spine questions on the OITE. This study looked to answer the following questions:

1. Have the most common subtopics of spine questions changed?;
2. How has the taxonomy of questions changed?;
3. Has there been a change to the most commonly referenced sources and the number of references utilized to support each question?;  
4. Has there been a change in the lag time of reference publication to examination date?;
5. Has there been a change in the frequency and type of imaging modalities provided?

It was hypothesized that there would be no change in subsections, taxonomy, reference sources, lag time, or use of imaging modalities between the two subsets.

Methods
A cross-sectional review of all OITE questions from 2009-2012 and 2017-2020 was performed to analyze question subtopic, taxonomy, references, lag time, and use of imaging modalities. Questions from 2009-2012 were obtained from past examinations that had been distributed by the AAOS. These questions were not delineated by content section, therefore two independent reviewers (*** & ***) determined which questions should be designated as spine. Any discrepancies were reviewed by a third author (***). Questions from 2017-2020 were obtained directly from the AAOS website via the ResStudy online portal.⁹ Question review was performed by first assessing the designated spine section and subsequently reviewing all other sections to identify spine questions that may have been categorized into a different section by the AAOS.  Questions from 2014-2016 examinations were not available for review as questions were not listed on the ‘Res Study’ online portal until 2017, and authors were unable to obtain copies of the examinations previously distributed by the AAOS.

All spine questions were further categorized into subtopics as previously defined by Farjoodi et al.¹⁰ Taxonomy of questions was determined based on the categorization system described by Buckwalter et al.¹¹ Taxonomy 1 (T1) questions tested isolated recognition and recall of specific information. Taxonomy 2 (T2) questions required the utilization of diagnostic data and imaging. Taxonomy 3 (T3) questions tested the application of knowledge or interpretation of information to solve a problem. The questions were categorized by two authors (*** & ***) and any disputes among taxonomy was determined by a third author (***).

The references provided to support each question were analyzed, noting the most cited references and total references cited per question. The length of time from article publication to appearance on the OITE exam was defined as lag time. The utilization of imaging modalities (e.g. Computed Tomography (CT), MRI (Magnetic Resonance Imaging), and Radiography (XR)) was recorded. 

Statistical analysis was performed using Excel (Microsoft, Washington) and Graphpad (Graphpad Holdings LLC, California). Continuous data between two groups were compared with independent 2-tailed T tests and 2-sample Z tests for proportions. Categorical data for groups were compared via chi square tests. Descriptive statistics are described as means with standard deviations and proportions.

Results
In review of 8 years of OITE questions included in this study (2009-2012; 2017-2020), there were a total of 202 spine questions (9.55%). Eight percent (6.5%-10.2%) of all questions from 2009-2012, and 11.22% (7.8%-16.7%) from 2017-2020, were categorized as spine and therefore included in our study (p=0.082) (Fig. 1).

Specific subtopics of individual spine questions were analyzed. Among questions from 2009-2012, the most tested subtopics included trauma (27.3%), postoperative complications (9.1%), and disc herniation (9.1%). From 2017-2020, the most tested subtopics were trauma (20.2%), degenerative disease (13.2%), and disc herniation (8.8%) (Table 1). There was a significant increase in questions relating to degenerative disease (p=0.037), while decreases in those related to trauma (p=0.238), disc herniation (p=0.936), and postoperative complications (p=0.430) were not significant.

SUBTOPIC	2009-2012	2017-2020
Trauma	24	23
Degenerative Disease	4	15
Disc Herniation	8	10
Postoperative Complications	8	7
Scoliosis	5	9
Anatomy	6	7
Spondylosis/Spondylolisthesis	4	7
Myelopathy	5	6
Infection	3	7
Low Back Pain	6	4
Tumor	4	4
Ankylosing Spondylitis	1	5
Diffuse Idiopathic Skeletal Hyperostosis (DISH)	0	1
Cauda Equina	0	1
Sports Clearance	1	0
Subjects Tested Once	9	8

For each spine question, taxonomy grading was assigned: T1 (35.1%), T2 (22.8%), and T3 (42.1%). Questions from 2009-2012 were categorized as T1 (35.2%), T2 (28.4%), and T3 (36.4%). Questions from 2017-2020 were categorized as T1 (35.1%), T2 (18.4%), and T3 (46.5%) (Fig. 2). Changes in proportions of question taxonomy were not significant for T1 (p=0.0984), T2 (p=0.093) or T3 (p=0.147). 

A total of 453 references were cited across 202 spine questions. The average number of references cited per question was 2.32 (+/- 0.59) in the early subset, and 2.14 ( +/- 0.48) in the later subset (p=0.008) (Fig. 3). From 2009-2012, the most referenced journals were Spine (29.3%), Journal of American Academy of Orthopedic Surgeons (JAAOS)  (13.3%), and Journal of Bone and Joint Surgery (JBJS) (10.6%). From 2017-2020, Spine (23.8%), JAAOS (10.2%), and JBJS (6.4%) were also the most referenced sources. Differences in Spine (p=0.190), JAAOS (p=0.308), and JBJS (p=0.105) were not significant (Fig. 4).

The lag time, as defined by time of reference publication to appearance on the OITE, was calculated. Spine questions from 2009-2012 had an average lag time for references was 7.84 years ( +/- 8.33), while questions from 2017-2020 had an average lag time of 6.30 years (SD +/- 7.40). This decrease in lag time was found to be significant (p=0.019) (Fig. 5).

Imaging (XR, CT, MRI, or Bone Scan) was provided for interpretation in 52.0% of spine questions. In the 2009-2012 examinations, 54.5% of spine questions provided at least one form of imaging; MRI (41.3%), XR (38.1%) CT (19.0%) and Bone Scan (1.6%). In 2017-2020, 50.0% of spine questions utilized at least one form of imaging; MRI (37.7%), XR (34.8%), CT (27.5%). There was no Bone Scan in the later subset. The decrease in questions that provided imaging was not found to be significant (p=0.522). The differences in proportion of MRI (p=0.674), XR (p=0.689) and CT (p=0.250) were not significant (Fig. 6). 

Discussion
With the recent collaboration between the ABOS and the AAOS to align the OITE with the Part I Certifying Exam, success on the OITE will only become more important in resident education. While prior studies have examined spine questions, their findings are no longer up to date.^10,12 Our study provides a more recent review of the spine section of the OITE, evaluating changes in questions subtopic, taxonomy, references and utilization of imaging modalities.

Traditionally, the AAOS OITE Blueprint demonstrates that spine questions should constitute 12% of total questions.⁹ In our study, spine questions constituted 8.0% of questions from 2009-2012 and 11.2% of questions from 2017-2020. However, the most recent examination reviewed (2020), had a much larger composition of spine questions (16.7%). Previous studies have found the spine section to make up between 8.4-9.0% of exam questions, which was consistent with our early subgroup.^10-12 This suggests a recent emphasis on knowledge of the spine for the OITE, demonstrated both by the AAOS OITE Blueprint and the data from our later subgroup.

In evaluating subtopics of questions, trauma was the most common subtopic within spine questions in both subgroups (2009-2012: 27.3%; 2017-2020: 20.2%), demonstrating its importance in success on this section. Farjoodi et al. found that 15.1% of questions from 2002-2007 examinations were related to trauma, which was the most common subtopic.¹⁰ Our study showed a significant increase in questions related to degenerative disease in recent exams (13.2%), which was found to be the most common subtopic (43.5%) in Agarwalla et al. review of spine OITE questions from 2013-2017.¹² When preparing for the spine section exam, extra attention should be focused on trauma and degenerative changes. 

In comparison of the taxonomy of spine questions of the two subgroups, there was an increase in T3 questions with an associated decrease in T2 questions. The increase in T3 questions continues a trend dating back to a prior review of questions from 2002-2007, which found T3 questions to compose 29.1% of spine questions.¹⁰ While simple recall of knowledge (T1) has remained constant, questions that involve the interpretation of data and imaging have shown a trend towards applying that information to solve a problem, necessitating a higher level of understanding. 

While there was an increase in the proportion of questions providing CT imaging, MRI remained the most utilized imaging modality in both subsets. MRI has previously been found to be the most common imaging modality by prior reviews of spine questions.^10,12 These prior studies demonstrate a trend in increased use of CT imaging, beginning at 9.5% as reported by Farjoodi et al. and 22.8% by Agarwalla et al.^10,12 

The top three most referenced journals remained the same between the two time periods: Spine, JAAOS, and JBJS. These journals made up over half of all citations for questions from 2009-2012, and 40% of questions from 2017-2020. Two prior reviews also found that Spine was the most referenced journal.^10,12 Lag time between article publication to citation on the OITE exam was found to decrease in more recent exam years, noting a trend towards more recent research. To the authors’ knowledge, no previous studies have evaluated lag time for references of spine questions. By reading recent spine literature from Spine, JBJS, and JAAOS, residents can optimize their utilization of journals in preparing for this section.

Limitations
This study has several limitations. We chose to analyze trends between two subsets of OITE questions from 2009-2012 and 2017-2020. As such, our findings may not be generalizable to all exams outside of these time intervals. Questions from 2017-2020 were obtained through the AAOS website, which divided the questions into subsections making spine questions easily identifiable.  However, it is possible that there could be an overlap in questions that included spine-related topics but were categorized into a different section by the AAOS. While all questions of different categories were also reviewed for the inclusion of spine questions, there may have been questions relating to the spine that were not included in our study. Questions from 2009-2012 examinations were obtained as PDF files of exams that had been previously distributed by the AAOS. These questions were not divided into content sections, and thus reviewer judgment was utilized to determine which questions would be designated as spine. While careful review of questions by two separate authors was performed, there may have been questions included in the study that would have been designated as an alternative section by the AAOS.

Likewise, there may have been spine questions that the reviewers considered to be of another subsection and thus were not included in our review. Assigning question taxonomy is subjective and could vary between reviewers, however each question taxonomy was verified by two residents with discrepancies further reviewed by third author. Buckwalter et al. found 85% agreement between test makers and residents when determining taxonomy.¹¹ Questions for OITE examinations between 2014-2016 were unable to be obtained, which limits the analysis of trends within our data points. 

Conclusion
In summary, this study analyzed recent trends on subsections, taxonomy, references, and use of imaging modalities within OITE spine questions. In the later subset, there was an increase in spine questions relating to degenerative disease. In evaluation of references provided to support each question, there were less citations per question, but a decrease in reference lag time. This manuscript can be used to guide residency program administration and residents to effectively design curricula to achieve success on the OITE, while providing a framework for future studies to evaluate changes to the OITE exam with the recent collaboration between the AAOS and ABOS to align the examinations. 

References

1. Luginbuhl JC, Sobel AD, Mulcahey MK. Analysis of the Sports Medicine Section of the Orthopaedic In-Training Examination: Improvements in Levels of Evidence and Question Taxonomy Over a 12-Year Period. Orthopedics. 2020 Sep 1;43(5):e460-e464. doi: 10.3928/01477447-20200619-02. Epub 2020 Jul 7. PMID: 32602922.
2. DePasse JM, Haglin J, Eltorai AEM, et al. Orthopedic In-Training Examination Question Metrics and Resident Test Performance. Orthop Rev (Pavia). 2017;9(2):7006. Published 2017 Jun 23. doi:10.4081/or.2017.7006
3. Swanson D, Marsh JL, Hurwitz S, et al. Utility of AAOS OITE scores in predicting ABOS Part I outcomes: AAOS exhibit selection. J Bone Joint Surg Am. 2013;95(12):e84. doi:10.2106/JBJS.L.00457
4. Herndon JH, Allan BJ, Dyer G, Jawa A, Zurakowski D. Predictors of success on the American Board of Orthopaedic Surgery examination. Clin Orthop Relat Res. 2009;467(9):2436-2445. doi:10.1007/s11999-009-0939-y
5. Klein GR, Austin MS, Randolph S, Sharkey PF, Hilibrand AS. Passing the Boards: can USMLE and Orthopaedic in-Training Examination scores predict passage of the ABOS Part-I examination? J Bone Joint Surg Am. 2004;86(5):1092-1095.
6. Dougherty PJ, Walter N, Schilling P, Najibi S, Herkowitz H. Do scores of the USMLE Step 1 and OITE correlate with the ABOS Part I certifying examination?: a multicenter study. Clin Orthop Relat Res. 2010;468(10):2797-2802. doi:10.1007/s11999-010-1327-3
7. Fritz E, Bednar M, Harrast J, et al. Do Orthopaedic In-Training Examination scores predict the likelihood of passing the American Board of Orthopaedic Surgery part I examination? An update with 2014 to 2018 data. J Am Acad Orthop Surg. 2021;29(24):e1370-e1377. doi:10.5435/JAAOS-D-20-01019
8. “The Diplomate E-Newsletter Fall 2021.” ABOS, 8 Dec. 2021, https://www.abos.org/the-diplomate-e-newsletter-fall-2021/. Accessed July 2022.
9. Orthopaedic In-Training Examination (OITE). https://www.aaos.org/education/about-aaos-products/orthopaedic-in-training-examination-oite/. Accessed February 2022.
10. Farjoodi P, Khanna AJ, Marker DR, et al. Evaluation of the Orthopaedic In-Training Examination: spine questions. J Surg Educ. 2010;67(3):139-142. doi:10.1016/j.jsurg.2010.04.001
11. Buckwalter JA, Schumacher R, Albright JP, et al. Use of an educational taxonomy for evaluation of cognitive performance. J Med Educ. 1981 Feb;56(2):115-21. doi: 10.1097/00001888-198102000-00006. PMID: 7463444.
12. Agarwalla A, Issa K, Changoor S, et al. Evaluation of Spine Questions on the Orthopaedic In-Training Examination: An Update from 2013 to 2017. J Long Term Eff Med Implants. 2018;28(4):297-302. doi: 10.1615/JLongTermEffMedImplants.2019030064. PMID: 31002620.

---

Required Disclosures and Declaration
Copyright Information: No Copyright Information Added
IRB Approval Information: Not applicable
Disclosure Information: No known conflicts of interest