Skip to content

Instantly share code, notes, and snippets.

@CatherineWong

CatherineWong/language-abstractions-clevr-artificial-icml-2021-documentation.md

Last active January 21, 2021 05:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save CatherineWong/edca3f941cb89fbbe163102a56bf214d to your computer and use it in GitHub Desktop.
Save CatherineWong/edca3f941cb89fbbe163102a56bf214d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
language-abstractions-clevr-artificial-icml-2021-documentation.md

The CLEVR Artificial Language learning environment uses the following lexicon and template classes:

Primitives, Constants, and Artificial Lexicon

Constants These are the basic constants that we would consider replacing with a single artificial term.

  • { "gray", "red", "blue", "green", "brown", "purple", "cyan", "yellow"}
  • {"cube", "sphere", "cylinder"}
  • {"small", "large"}
  • {"rubber", "metal"}
  • {1, 2, 3, 4, 5, 6, 7, 8, 9, 10} -> These are preserved in numerical form. They are not replaced with a term.

Primitives These are the basic functions that we would consider replacing with a single artificial term.

  • [(What) if you removed all of the ____] -> | e.g. [What if you removed all the] large metal spheres?; [If you removed the red things], how many cubes would be left?

  • [(What) if the ______ became a _____?] | e.g. [What if the] big red cube [became a] small yellow sphere? ; [If all the (small green cubes) became (rubber)], how many rubber things would there be?

  • [How many ______] | e.g. What number of small rubber cylinders are there?

  • [What shape is the ], [What material is the ], [What color is the ], [What size is the ] ; this is a query shape primitive. It can be used in the 'same' filters as well.

  • [ in front of the ], [ to the left of the ], [ to the right of the ], [ behind the ]

  • [____ or ____]

  • [___of the same ____ as the ____]

  • [greater than], [less than]

Our syntax is closely based on a flattened traversal of the program syntax tree, in which functions are prefixed to their arguments. For readability we use a comma delimiter that can be interpreted similar to a statement delimiter, in which the previous statement was assigned to a variable and its return value is now passed to the next statement. (E.g'If you removed all of the red things, then how many things would there be?') Binary functions use the following transitive form: (e..g Transform X into Y becomes X TRANSFORM Y)

Templates.

We write translations and examples for templates using an english CONSTANT to indicate a single artificial token for readability. Samples are then shown, but samples are not consistent for artificial tokens in this gist.

Object localization

2_localization_train, 2_localization_val: Contains object localization questions with 1 or more chained filters.

  • Find the <Z> <C> <M> <S>. Each of the constant terms is interpreted as automatically including a filter term. We translate this simply as <Z> <C> <M> <S>, e.g. Find the big red cubes. -> BIG RED CUBE

transformationCLEVR

2_remove

  • (x1) What if you removed all of the <Z> <C> <M> <S>s? | REMOVE <Z> <C> <M> <S>
  • (x1) If you removed the <Z> <C> <M> <S>s, how many things would be left? | REMOVE <Z> <C> <M> <S>s?, COUNT
  • (x1) If you removed the <C> things, how many <S>s would be left? | ObjectSet -> Int | REMOVE <Z> <C> <M> <S>s?, COUNT <S>
  • (x1) If you removed the <S>s, how many <Z> things would be left? | ObjectSet -> Int | REMOVE <S>, COUNT <Z>
  • (x1) If you removed the <C> <S>s, how many <S>s would be left? | ObjectSet -> Int | REMOVE <C> <S>, COUNT <S>

2_transform Contains 6 templates.

  • (x1) What if the <Z> <C> <M> <S> became a <Z2> <C2> <M2> <S2>? | ObjectSet -> ObjectSet | <Z> <C> <M> <S> TRANSFORM <Z2> <C2> <M2> <S2>
  • (x1) What if all the <Z> <C> <M> <S>s became <Z2> <C2> <M2> <S2>s? | ObjectSet -> ObjectSet | <Z> <C> <M> <S> TRANSFORM <Z2> <C2> <M2> <S2>
  • (x1) If all of the <Z> <C> <M> <S>s became <C2>, how many <C2> things would there be? | ObjectSet -> Int | <Z> <C> <M> <S> TRANSFORM <C2>, COUNT <C2>
  • (x1) "If all of the <Z> <C> <M> <S>s became <M2>, how many <M2> things would there be?| ObjectSet -> Int |<Z> <C> <M> <S> TRANSFORM <M2>, COUNT <M2>
  • (x1) "If all of the <Z> <C> <M> <S>s became <C2> <S2>s, how many <C2> <S2>s would there be? | ObjectSet -> Int | <Z> <C> <M> <S> TRANSFORM <C2> <S2>, COUNT <C2> <S2>
  • (x1) If all of the <Z> <C> <M> <S>s became <Z2>, how many <Z> things would there be? | ObjectSet -> Int | <Z> <C> <M> <S> TRANSFORM <Z2>, COUNT <Z>

miniCLEVR

1_zero_hop :

  • (x1) How many <Z> <C> <M> <S>s are there? | ObjectSet -> Int | COUNT <Z> <C> <M> <S>s

  • (x1) What shape is the <Z> <C> <M> <S>? | ObjectSet -> Shape | GET_SHAPE <Z> <C> <M> <S>

  • (x1) What material is the <Z> <C> <M> <S>? | ObjectSet -> Material | GET_MATERIAL <Z> <C> <M> <S>

  • (x1) What color is the <Z> <C> <M> <S>? | ObjectSet -> Color | GET_COLOR <Z> <C> <M> <S>

  • (x1) What size is the <Z> <C> <M> <S>? | ObjectSet -> Size | GET_SIZE <Z> <C> <M> <S>

    1_one_hop

  • (x1) What number of <Z2> <C2> <M2> <S2>s are <R> the <Z> <C> <M> <S>? | ObjectSet -> Int | COUNT <Z2> <C2> <M2> <S2> RELATION_OF <Z> <C> <M> <S>

  • (x1) What size is the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | ObjectSet -> Size | GET_SIZE <Z2> <C2> <M2> <S2> RELATION_OF <Z> <C> <M> <S>

  • (x1) What color is the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | ObjectSet -> Color | GET_COLOR <Z2> <C2> <M2> <S2> RELATION_OF <Z> <C> <M> <S>

  • (x1) What is the material of the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | ObjectSet -> Material | GET_MATERIAL <Z2> <C2> <M2> <S2> RELATION_OF <Z> <C> <M> <S>

  • (x1) What shape is the <Z2> <C2> <M2> <S2> [that is] <R> the <Z> <C> <M> <S>? | ObjectSet -> Shape | GET_SHAPE <Z2> <C2> <M2> <S2> RELATION_OF <Z> <C> <M> <S>

1_single_or - (x1) How many things are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | COUNT <Z> <C> <M> <S> OR <Z2> <C2> <M2> <S2>

  • (x1) How many <Z3> things are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | <Z> <C> <M> <S> OR <Z2> <C2> <M2> <S2>, COUNT <Z3>
  • (x1) How many <C3> things are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | <Z> <C> <M> <S> OR <Z2> <C2> <M2> <S2>, COUNT <C3>
  • (x1) How many <M3> things are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | <Z> <C> <M> <S> OR <Z2> <C2> <M2> <S2>, COUNT <M3>
  • (x1) How many <S3>s are [either] <Z> <C> <M> <S>s or <Z2> <C2> <M2> <S2>s? | ObjectSet -> Int | <Z> <C> <M> <S> OR <Z2> <C2> <M2> <S2>, COUNT <S3>

1_same_relate_restricted : Contains questions that require finding another object with the same attribute as a query object.

  • (x1) How many other things are the same size as the <Z> <C> <M> <S>? | ObjectSet -> Int | GET_SIZE SAME <Z> <C> <M> <S>, COUNT
  • (x1) How many other objects are there of the same color as the rubber sphere? | ObjectSet -> Int | GET_COLOR SAME <Z> <C> <M> <S>, COUNT
  • (x1) How many other <Z2> <C2> <M2> <S2>s have the same material as the <Z> <C> <M> <S>? | ObjectSet -> Int | GET_MATERIAL SAME <Z> <C> <M> <S>, COUNT <Z2> <C2> <M2> <S2>
  • (x1) How many other <Z2> <C2> <M2> <S2>s have the same shape as the <Z> <C> <M> <S>? | ObjectSet -> Int | GET_SHAPE SAME <Z> <C> <M> <S>, COUNT <Z2> <C2> <M2> <S2>
  • (x1) What shape is the other <Z2> <C2> <M2> <S2> that is the same size as the <Z> <C> <M> <S>? | ObjectSet -> Size | GET_SIZE SAME <Z> <C> <M> <S>, GET_SHAPE <Z2> <C2> <M2> <S2>
  • (x1) What material is the other <Z2> <C2> <M2> <S2> that is the same color as the <Z> <C> <M> <S>? | ObjectSet -> Color | GET_COLOR SAME <Z> <C> <M> <S>, GET_MATERIAL <Z2> <C2> <M2> <S2>

1_compare_integer

  • (x1) Are there fewer <Z> <C> <M> <S>s than <Z2> <C2> <M2> <S2>s? | ObjectSet -> Bool | COUNT <Z> <C> <M> <S> LESS_THAN COUNT <Z2> <C2> <M2> <S2>
  • (x1) Are there more <Z> <C> <M> <S>s than <Z2> <C2> <M2> <S2>s? | ObjectSet -> Bool | COUNT <Z> <C> <M> <S> GREATER_THAN COUNT <Z2> <C2> <M2> <S2>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment