{"id":1065,"date":"2024-04-24T14:47:18","date_gmt":"2024-04-24T18:47:18","guid":{"rendered":"https:\/\/journal.aoao.org\/?p=1065"},"modified":"2024-04-24T14:52:39","modified_gmt":"2024-04-24T18:52:39","slug":"how-does-a-large-language-model-llm-artificial-intelligence-ai-fare-with-adult-reconstruction-knowledge","status":"publish","type":"post","link":"https:\/\/journal.aoao.org\/?p=1065","title":{"rendered":"How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge?"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><sup>1<\/sup>Lum Z,\u00a0<sup>1<\/sup>Collins D,\u00a0<sup>1<\/sup>Guntupalli L,\u00a0<sup>1<\/sup>Dennison S,\u00a0<sup>2<\/sup>Choudhary S,\u00a0<sup>2<\/sup>Saiz A,\u00a0<sup>2<\/sup>Randall R<br><sup>1<\/sup>Nova Southeastern University, Fort Lauderdale, FL, United States;\u00a0<sup>2<\/sup>University of California Davis, Sacramento, CA, United states<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Introduction<\/strong><br>As the capabilities of artificial intelligence (AI) continue to advance, it is important to regularly evaluate competency to maintain high standards and preventing potential errors or biases, that could deliver misinformation that could harm patients or spread inaccurate information. A new AI model using large language models (LLM) and non-specific domain areas has gained recent attention in its novel way to process information. We wanted to test its performance to correctly answer hip &amp; knee arthroplasty questions compared to other subject types and taxonomy question type (recall, interpretation, knowledge application).<br><br><strong>Methods<\/strong><br>We asked ChatGPT, 3173 questions based on the Orthopaedic In-Training Exam (OITE) and 757 questions from the real OITE. Questions were categorized by subject type, and by taxonomy type. These questions were then entered into the AI chatbot and score was recorded. Multivariate logistic regression analysis was performed comparing hip &amp; knee arthroplasty questions with other question types, and based upon taxonomy.<br><br><strong>Results<\/strong><br>After exclusions, ChatGPT answered 960\/1871 (51%) of total questions correctly and 77\/194 (40%) of hip &amp; knee arthroplasty questions correctly, which was in the lower performing subject types. Reconstruction testing exhibited poorer performance than Basic Science (p&lt;0.001, 3.23 OR [2.26-4.65]), Pathology\/Oncology (p&lt;0.001, 2.82 OR [1.72-4.65], Knee &amp; Sports Medicine (p&lt;0.001, OR 1.99 [1.34-2.96]) and Pediatrics (p=0.011, OR 1.70 [1.12-2.56]). When evaluating sub-group taxonomy analysis, univariate logistic regression demonstrated the AI\u2019s lower performance in taxonomy type 3 compared to type 1 (50% vs 41%, p=0.049).<br><br><strong>Discussion<\/strong><br>The AI LLM may not be effective in answering orthopaedic questions related to hip and knee arthroplasty. Furthermore, the study\u2019s taxonomy analysis highlights the importance of considering the question structure when evaluating AI performance. Ultimately, as AI continues to evolve and advance, it will be important to consider its limitations and potential biases to ensure its responsible and ethical use.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/journal.aoao.org\/wp-content\/uploads\/2024\/04\/AR-9-Image-1.jpg\" target=\"_blank\" rel=\"noreferrer noopener\">Image Number: 1<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/journal.aoao.org\/wp-content\/uploads\/2024\/04\/AR-9-Image-2.jpg\" target=\"_blank\" rel=\"noreferrer noopener\">Image Number: 2<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/journal.aoao.org\/wp-content\/uploads\/2024\/04\/AR-9-Image-3.jpg\" target=\"_blank\" rel=\"noreferrer noopener\">Image Number: 3<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>1Lum Z,\u00a01Collins D,\u00a01Guntupalli L,\u00a01Dennison S,\u00a02Choudhary S,\u00a02Saiz A,\u00a02Randall R1Nova Southeastern University, Fort Lauderdale, FL, United States;\u00a02University of California Davis, Sacramento, CA, United states IntroductionAs the capabilities of artificial intelligence (AI) continue to advance, it is important to regularly evaluate competency to maintain high standards and preventing potential errors or biases, that could deliver misinformation that could [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41],"tags":[40,60],"class_list":["post-1065","post","type-post","status-publish","format-standard","hentry","category-adult-reconstruction","tag-abstracts","tag-spring-2024"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge? - AOAO Journal<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/journal.aoao.org\/?p=1065\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge? - AOAO Journal\" \/>\n<meta property=\"og:description\" content=\"1Lum Z,\u00a01Collins D,\u00a01Guntupalli L,\u00a01Dennison S,\u00a02Choudhary S,\u00a02Saiz A,\u00a02Randall R1Nova Southeastern University, Fort Lauderdale, FL, United States;\u00a02University of California Davis, Sacramento, CA, United states IntroductionAs the capabilities of artificial intelligence (AI) continue to advance, it is important to regularly evaluate competency to maintain high standards and preventing potential errors or biases, that could deliver misinformation that could [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/journal.aoao.org\/?p=1065\" \/>\n<meta property=\"og:site_name\" content=\"AOAO Journal\" \/>\n<meta property=\"article:published_time\" content=\"2024-04-24T18:47:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-04-24T18:52:39+00:00\" \/>\n<meta name=\"author\" content=\"Ray Hall\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ray Hall\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/journal.aoao.org\/?p=1065\",\"url\":\"https:\/\/journal.aoao.org\/?p=1065\",\"name\":\"How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge? - AOAO Journal\",\"isPartOf\":{\"@id\":\"https:\/\/journal.aoao.org\/#website\"},\"datePublished\":\"2024-04-24T18:47:18+00:00\",\"dateModified\":\"2024-04-24T18:52:39+00:00\",\"author\":{\"@id\":\"https:\/\/journal.aoao.org\/#\/schema\/person\/b3ff623305ba59e337e22c31c9fc42e7\"},\"breadcrumb\":{\"@id\":\"https:\/\/journal.aoao.org\/?p=1065#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/journal.aoao.org\/?p=1065\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/journal.aoao.org\/?p=1065#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/journal.aoao.org\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/journal.aoao.org\/#website\",\"url\":\"https:\/\/journal.aoao.org\/\",\"name\":\"AOAO Journal\",\"description\":\"The Journal of the American Osteopathic Academy of Orthopedics (JAOAO)\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/journal.aoao.org\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/journal.aoao.org\/#\/schema\/person\/b3ff623305ba59e337e22c31c9fc42e7\",\"name\":\"Ray Hall\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/journal.aoao.org\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/77c2e3f00618d6950c250a915344c15d9f601fb38bd861cae3882e3489cd7d18?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/77c2e3f00618d6950c250a915344c15d9f601fb38bd861cae3882e3489cd7d18?s=96&d=mm&r=g\",\"caption\":\"Ray Hall\"},\"url\":\"https:\/\/journal.aoao.org\/?author=3\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge? - AOAO Journal","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/journal.aoao.org\/?p=1065","og_locale":"en_US","og_type":"article","og_title":"How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge? - AOAO Journal","og_description":"1Lum Z,\u00a01Collins D,\u00a01Guntupalli L,\u00a01Dennison S,\u00a02Choudhary S,\u00a02Saiz A,\u00a02Randall R1Nova Southeastern University, Fort Lauderdale, FL, United States;\u00a02University of California Davis, Sacramento, CA, United states IntroductionAs the capabilities of artificial intelligence (AI) continue to advance, it is important to regularly evaluate competency to maintain high standards and preventing potential errors or biases, that could deliver misinformation that could [&hellip;]","og_url":"https:\/\/journal.aoao.org\/?p=1065","og_site_name":"AOAO Journal","article_published_time":"2024-04-24T18:47:18+00:00","article_modified_time":"2024-04-24T18:52:39+00:00","author":"Ray Hall","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Ray Hall","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/journal.aoao.org\/?p=1065","url":"https:\/\/journal.aoao.org\/?p=1065","name":"How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge? - AOAO Journal","isPartOf":{"@id":"https:\/\/journal.aoao.org\/#website"},"datePublished":"2024-04-24T18:47:18+00:00","dateModified":"2024-04-24T18:52:39+00:00","author":{"@id":"https:\/\/journal.aoao.org\/#\/schema\/person\/b3ff623305ba59e337e22c31c9fc42e7"},"breadcrumb":{"@id":"https:\/\/journal.aoao.org\/?p=1065#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/journal.aoao.org\/?p=1065"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/journal.aoao.org\/?p=1065#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/journal.aoao.org\/"},{"@type":"ListItem","position":2,"name":"How does a Large Language Model (LLM) Artificial Intelligence (AI) Fare with Adult Reconstruction Knowledge?"}]},{"@type":"WebSite","@id":"https:\/\/journal.aoao.org\/#website","url":"https:\/\/journal.aoao.org\/","name":"AOAO Journal","description":"The Journal of the American Osteopathic Academy of Orthopedics (JAOAO)","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/journal.aoao.org\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/journal.aoao.org\/#\/schema\/person\/b3ff623305ba59e337e22c31c9fc42e7","name":"Ray Hall","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/journal.aoao.org\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/77c2e3f00618d6950c250a915344c15d9f601fb38bd861cae3882e3489cd7d18?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/77c2e3f00618d6950c250a915344c15d9f601fb38bd861cae3882e3489cd7d18?s=96&d=mm&r=g","caption":"Ray Hall"},"url":"https:\/\/journal.aoao.org\/?author=3"}]}},"featured_image_src":null,"featured_image_src_square":null,"author_info":{"display_name":"Ray Hall","author_link":"https:\/\/journal.aoao.org\/?author=3"},"_links":{"self":[{"href":"https:\/\/journal.aoao.org\/index.php?rest_route=\/wp\/v2\/posts\/1065","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/journal.aoao.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/journal.aoao.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/journal.aoao.org\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/journal.aoao.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1065"}],"version-history":[{"count":0,"href":"https:\/\/journal.aoao.org\/index.php?rest_route=\/wp\/v2\/posts\/1065\/revisions"}],"wp:attachment":[{"href":"https:\/\/journal.aoao.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1065"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/journal.aoao.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1065"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/journal.aoao.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1065"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}