A dataset with 52 Text to image model evaluation [P]

I created a simple text to image benchmark.

I curated 192 prompts that are difficult for T2I models in various ways: text rendering, spatial reasoning, human realism, negations, etc...

I then asked a VLM to judge every output against a pre-specified binary question with the ground truth baked in.

I'm publishing all the results including the images. (Most public T2I leaderboards don't publish the actual images and that's a shame IMO)

There is currently 52 model tested! more than 9k images have been generated and analysed!

Full methodology: https://imagebench.ai/methodology-v1

Hugging face dataset: https://huggingface.co/datasets/dh7/imagebench (it contains the prompts to reproduce the results AND the results)

Github: https://github.com/dh7/image-bench-ai

Gallery to inspect the results: https://imagebench.ai/gallery

Leaderboard: https://imagebench.ai/imagebench-v1

Limitations: it's text to image only, and VLM are not perfect as a judge.

Let me know what could be useful from there!