Evaluating AI Vision: A Spider Photo Benchmark Reveals Accuracy Gaps

I tested nine vision models on the same 2,000 spider photos. The highest exact-species accuracy was 49.85%. The tasks, predictions, scoring code, and run settings are public.

The setup:

• Data: 2,000 photos covering 671 species and subspecies, sampled from a filtered collection of research-grade iNaturalist observations. The species list came from a Polish checklist; the photos were taken worldwide.
• Task: Pick one of 20 scientific names. Each list contains the expected species, up to nine alternatives from the same family, and other-family names to fill the remaining places.
• Controls: Every model received the same prepared image bytes and the same candidate lists in the same order. Sampling and candidate selection used fixed seeds.
• Scoring: Matches against the iNaturalist labels, divided by all 2,000 assigned photos. Failed, missing, and invalid answers count as incorrect.

These are the nine high-effort runs used in my write-up:

The top two were separated by 47 photos. Additional runs at other effort settings are included in the repository.

I also grouped predictions by taxonomy. Gemini’s answers matched the expected genus 63.65% of the time and the expected family 93.40% of the time, including exact-species matches in both figures.

Code, frozen tasks, predictions, and results

Benchmark protocol

My write-up with charts and example predictions