Skip to content

Instantly share code, notes, and snippets.

@CatherineWong

CatherineWong/language-abstractions-clevr-icml-2021-documentation.md

Last active January 31, 2021 04:34
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save CatherineWong/8e26a0787c96a74a877d0dba8e2ea3e4 to your computer and use it in GitHub Desktop.
Save CatherineWong/8e26a0787c96a74a877d0dba8e2ea3e4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
language-abstractions-clevr-icml-2021-documentation.md

The CLEVR ICML dataset uses the following template classes:

Object localization

2_localization_train: Contains simple object localization questions with basic filter queries. Contains 7 templates.

  • (x1) Find the <Z> things. ObjectSet -> ObjectSet | Find the big things. Find the small things.
  • (x1)Find the <C> things. ObjectSet -> ObjectSet | Find the red things. Find the blue things.
  • (x1)Find the <M> things. ObjectSet -> ObjectSet | Find the rubber things.
  • (x1) Find the <S>s. ObjectSet -> ObjectSet | Find the cubes.
  • (x3)Find the <Z> <C> <M> <S>. ObjectSet -> ObjectSet | Find the big red cubes. Find the small blue rubber things. Generated 58 training questions.

2_localization_val: Contains object localization questions with only chained filters. Contains 3 templates.

  • (x3)Find the <Z> <C> <M> <S>. ObjectSet -> ObjectSet | Find the big red cubes. Find the small blue rubber things. Generated 30 testing questions.

transformationCLEVR

2_remove : Contains questions that require using the remove primitive. Contains 5 templates.

  • (x1) What if you removed all of the <Z> <C> <M> <S>s? | ObjectSet -> ObjectSet | What if you removed all the large metal spheres?
  • (x1) If you removed the <Z> <C> <M> <S>s, how many things would be left? | ObjectSet -> Int | If you removed all the small yellow cubes, how many things would be left?
  • (x1) If you removed the <C> things, how many <S>s would be left? | ObjectSet -> Int | If you removed the red things, how many cubes would be left?
  • (x1) If you removed the <S>s, how many <Z> things would be left? | ObjectSet -> Int | If you removed the spheres, how many big things would be left?
  • (x1) If you removed the <C> <S>s, how many <S>s would be left? | ObjectSet -> Int | If you removed the red cubes, how many cubes would be left? Generated 23 training questions (5x) per question; 9 test questions (2x per question).

2_transform : Contains questions that require using some of the transform primitives. Contains 6 templates.

  • (x1) What if the <Z> <C> <M> <S> became a <Z2> <C2> <M2> <S2>? | ObjectSet -> ObjectSet | What if the big red cube became a small yellow sphere?
  • (x1) What if all the <Z> <C> <M> <S>s became <Z2> <C2> <M2> <S2>s? | ObjectSet -> ObjectSet | What if all the small metal things became blue cubes?
  • (x1) If all of the <Z> <C> <M> <S>s became <C2>, how many <C2> things would there be? | ObjectSet -> Int | If all the red rubber things became green, how many green things would there be?
  • (x1) "If all of the <Z> <C> <M> <S>s became <M2>, how many <M2> things would there be?| ObjectSet -> Int | If all the small green cubes became rubber, how many rubber things would there be?
  • (x1) "If all of the <Z> <C> <M> <S>s became <C2> <S2>s, how many <C2> <S2>s would there be? | ObjectSet -> Int | If all the small green cubes became red spheres, how many red spheres would there be?
  • (x1) If all of the <Z> <C> <M> <S>s became <Z2>, how many <Z> things would there be? | ObjectSet -> Int | If all of the small green cubes became red, how many red things would there be? Generated 22 training questions (5x) per question; 10 test questions (2x per question).

metaCLEVR

This is an experimental dataset of questions that involve reasoning about the attributes of a set of filtered objects. 2_meta_zero_hop

  • "How many different shapes of <Z> <C> <M> s are there? | ObjectSet -> Int | How many different shapes of blue rubber things are there?
  • "How many different colors of <Z> <M> <S> s are there? | ObjectSet -> Int | How many different colors of cubes are there?
  • "How many different sizes of <C> <M> <S> s are there? | ObjectSet -> Int | How many different sizes of red metal cylinders are there?
    • "How many different materials of <C> <Z> <S> s are there? | ObjectSet -> Int | How many different materials of big blue things are there?

2_meta_unique - "Do the <Z> <C> <M> <S> come in a unique {color | shape | size | material }? | ObjectSet -> Boolean | Do the red cubes come in unique size? Do the cylinders come in a unique material?

2_meta_one_hop These questions compose one of the question types below over the attribute properties

  • Are there more unique {colors | shapes | sizes | materials} than there are unique {colors | shapes | sizes | materials }? | ObjectSet -> Boolean | Are there more unique colors than there are unique shapes?
  • If I removed all of the <Z> <C> <M> <S>, how many unique {colors | shapes | sizes | materials } of objects would their be left? | ObjectSet -> Int | If I removed all of the blue cubes, how many unique sizes of objects would be left? -How many unique {colors | shapes | sizes | materials } are there for objects that are either <Z> <C> <M> <S> or <Z2 <C2> <M2> <S2> ? | ObjectSet -> Int | How many unique colors are there for objects that are either big cubes or big cylinders?

miniCLEVR

We generate questions with 5 I/O examples each.

1_zero_hop : Contains questions that require directly evaluating some query about a set of filters. Contains 5 templates.

  • (x1) How many <Z> <C> <M> <S>s are there? | ObjectSet -> Int | What number of small rubber cylinders are there?
  • (x1) What shape is the <Z> <C> <M> <S>? | ObjectSet -> Shape | What is the shape of the purple thing?
  • (x1) What material is the <Z> <C> <M> <S>? | ObjectSet -> Material | What is the small sphere made of?
  • (x1) What color is the <Z> <C> <M> <S>? | ObjectSet -> Color | What is the color of the large sphere?
  • (x1) What size is the <Z> <C> <M> <S>? | ObjectSet -> Size | The rubber cube has what size? We generate 30 training questions (6x template); 30 testing questions (3x template).

1_one_hop : Contains questions that require evaluating a relation before evaluating a query. Contains 5 templates.

  • (x1) What number of <Z2> <C2> <M2> <S2>s are <R> the <Z> <C> <M> <S>? | ObjectSet -> Int | There is a small purple rubber thing; what number of rubber cylinders are to the right of it?
  • (x1) What size is the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | ObjectSet -> Size |There is a metal cylinder right the small purple metal thing; what is its size?
  • (x1) What color is the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | ObjectSet -> Color | What color is the metal thing left of the gray thing?
  • (x1) What is the material of the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | ObjectSet -> Material | There is a large thing behind the small gray thing; what is its material?
  • (x1) What shape is the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | ObjectSet -> Shape | There is a large thing in front of the small metal cube; what is its shape?

We generate 30 training questions (6x template); 10 testing questions (2x template).

1_single_or : Contains questions that require evaluating an 'or' query. Contains 5 templates.

  • (x1) How many things are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | How many things are large metal cylinders or small gray things?
  • (x1) How many <Z3> things are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | What number of small objects are either rubber cylinders or green metal things?
  • (x1) How many <C3> things are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | How many red objects are either large metal things or small rubber cylinders?
  • (x1) How many <M3> things are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | How many rubber things are either small spheres or large purple things?
  • (x1) How many <S3>s are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | What number of spheres are small cyan metal things or small rubber things?

*** We generate 25 training questions (5x template); 10 testing questions (2x template). ***

1_same_relate_restricted : Contains questions that require finding another object with the same attribute as a query object. Contains 6 templates.

  • (x1) How many other things are the same size as the <Z> <C> <M> <S>? | ObjectSet -> Int | What number of other things are there of the same size as the metal sphere?
  • (x1) How many other objects are there of the same color as the rubber sphere? | ObjectSet -> Int | How many other objects are there of the same color as the rubber sphere?
  • (x1) How many other <Z2> <C2> <M2> <S2>s have the same material as the <Z> <C> <M> <S>? | ObjectSet -> Int | What number of other spheres are made of the same material as the large sphere?
  • (x1) How many other <Z2> <C2> <M2> <S2>s have the same shape as the <Z> <C> <M> <S>? | ObjectSet -> Int | What number of large metal things have the same shape as the small red metal thing?
  • (x1) What shape is the other <Z2> <C2> <M2> <S2> that is the same size as the <Z> <C> <M> <S>? | ObjectSet -> Size | What shape is the brown thing that is the same size as the gray rubber thing?
  • (x1) What material is the other <Z2> <C2> <M2> <S2> that is the same color as the <Z> <C> <M> <S>? | ObjectSet -> Color | What is the material of the small cylinder that is the same color as the metal cube?

*** We generate 18 training questions (3x template); 12 testing questions (2x template). ***

1_compare_integer : Contains boolean qusetions that require explicitly comparing the counts of different sets of objects. Contains 2 templates.

  • (x1) Are there fewer <Z> <C> <M> <S>s than <Z2> <C2> <M2> <S2>s? | ObjectSet -> Bool | Is the number of large rubber cubes less than the number of large green rubber things?
  • (x1) Are there more <Z> <C> <M> <S>s than <Z2> <C2> <M2> <S2>s? | ObjectSet -> Bool | Is the number of purple things greater than the number of small cubes?

*** We generate 6 training questions (3x template); 4 testing qusetions (2x template). ***

The CLEVR ICML dataset uses the following starting DSLs:

CLEVR Original.

As reference, we provide a reimplementation of the 'original' CLEVR primitives used to generate the dataset itself, along with a additional 'map' and 'transform' primitives required to handle all of the 'scene transformation' questions in principle. These primitives are: ` [clevr_relate] # Get spatial relations. (e.g. 'objects to the left')

+ [clevr_query_color,clevr_query_size, clevr_query_material, clevr_query_shape] # Get attribute constants

+ [clevr_eq_color, clevr_eq_size, clevr_eq_material, clevr_eq_shape]  # Test equality of attribute constants

+ [clevr_union, clevr_intersect, clevr_difference]  # Set operations

+ [clevr_count, clevr_is_gt, clevr_is_lt] 

+ [clevr_unique] 

+ [clevr_filter_material, clevr_filter_size, clevr_filter_color, clevr_filter_shape] 

+ [clevr_same_material, clevr_same_size, clevr_same_color, clevr_same_shape]

and [clevr_map] + [clevr_transform_color, clevr_transform_material, clevr_transform_size, clevr_transform_shape]`

CLEVR bootstrap.

These primitives include list manipulation functions intended to allow the model to bootstrap many of the key primitives in the CLEVR dataset; in particular, we eliminate the filter and same primitives. We do also provide additional 'map' and 'transform' primitives required to handle all of the 'scene transformation' questions in principle. These primitives are: ` [clevr_relate] # Get spatial relations. (e.g. 'objects to the left')

+ [clevr_query_color,clevr_query_size, clevr_query_material, clevr_query_shape] # Get attribute constants

+ [clevr_eq_color, clevr_eq_size, clevr_eq_material, clevr_eq_shape] # Test equality of attribute constants

+ [clevr_union, clevr_intersect, clevr_difference] # Set operations

+ [clevr_count, clevr_is_gt, clevr_is_lt] \ # Count, >, <

+ [clevr_car, clevr_if, clevr_empty, clevr_fold, clevr_add] # List transformations.

and [clevr_map] + [clevr_transform_color, clevr_transform_material, clevr_transform_size, clevr_transform_shape]

`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment