Welcome to the GenAI Painting Evaluation Study!

This survey is intended for research purposes and involves ranking the selected paintings from the Getty Collection. These rankings will serve as a baseline to evaluate the performance of Large Language Models on the same tasks.

Context:

You are a painting evaluator. You will be presented with 20 pages, each consisting only of the textual descriptions of one painting and 10 form statements. Your task is to rank how well the painting's textual description aligns with the statements (e.g., "The description includes references to nature, such as landscapes."). Evaluate each painting individually based on the presented statements, without comparing it to other paintings. Images of paintings are not available as we currently use only textual descriptions for our validation process.

Your task:

Rank each painting from 1 to 5 based on how well it meets the form statements:

  1. Does not meet the criteria or relevant information is missing from the description.
  2. Meets very few aspects.
  3. Meets some aspects.
  4. Meets most aspects.
  5. Meets entirely the criteria.
If the information needed to answer a question is missing from the painting's metadata or description (e.g., description, place of creation, or other relevant details), assign a rank of 1.

Note 1: You will see one textual description of a painting per page. Please rank all 10 statements before continuing.
Note 2: Each description consists mostly of metadata information followed by the content description of the painting, while a few of them involve metadata only. The metadata and the description appear as separate paragraphs.
Note 3: The content description of the painting is entirely copied from the Getty Collection website. Some text may appear unstructured due to the formatting.

For any questions, please write to: i.vasic@pm.univpm.it

Enjoy the art and the survey!