Comparing skin cancer AI systems
This skin cancer AI test compared GPT-6 Astra, Claude Fable 5.1, and a fine-tuned Inception v3 convolutional neural network. The researcher randomly selected 200 lesions from the public MILK10k dataset, all confirmed by histopathology, and asked each system to choose among six diagnoses and recommend discharge, biopsy, or follow-up.
The CNN gave the correct first-choice diagnosis for 125 cases, or 62%; the two LLMs scored 84 and 85 cases, both 42%. When a second choice was allowed, the CNN covered 173 cases, compared with 134 and 138 for the LLMs. It was generally stronger on melanoma and benign lesions, although it did not lead in every category.
Why a specialized model still led
The CNN was fine-tuned on public dermatology images and combined clinical photographs with dermoscopic views; the test lesions and other images from the same patients were excluded before training. Its architecture dates to 2015 and it was trained for roughly twenty minutes, yet it can run in a phone browser. The result suggests that a specialized model may fit a focused visual task better than a general language model.
Key limits of the skin cancer AI test
The result should not be generalized directly to clinical care. The sample contained only 200 lesions, including 30 melanomas, and each LLM case was run once, so its output may vary. More importantly, at its preset threshold the CNN discharged 18 of the 30 melanomas, despite its higher overall diagnostic accuracy. The researcher also notes that this is only a baseline architecture; stronger dermatology systems use larger datasets, vision transformers, or ensembles. When prompted as patients, the LLMs advised seeing a doctor in nearly every case, but caution is not the same as diagnostic accuracy. The study is therefore best read as evidence about model choice and evaluation design, not as support for replacing dermatologists with an old network.