Skip to content

Instantly share code, notes, and snippets.

@ioquatix
Created April 19, 2017 03:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ioquatix/bddda36d9e4ffaceb7a62d7b62259121 to your computer and use it in GitHub Desktop.
Save ioquatix/bddda36d9e4ffaceb7a62d7b62259121 to your computer and use it in GitHub Desktop.
user_event schema
Table "public.user_event"
Column | Type | Modifiers
-------------+-----------------------------+---------------------------------------------------------
id | integer | not null default nextval('user_event_id_seq'::regclass)
user_id | integer | not null
latitude | numeric(9,6) |
longitude | numeric(9,6) |
active | boolean | not null default true
poi_id | integer |
deal_id | integer |
category_id | integer |
what | character varying(32) | not null
locale | character varying(8) |
created_at | timestamp without time zone | not null
parameters | jsonb |
Indexes:
"user_event_pkey" PRIMARY KEY, btree (id)
"index_user_event_for_calls" btree (poi_id, created_at) WHERE what::text = 'profile_call'::text
"index_user_event_for_nearby_polls" btree (created_at, latitude, longitude) WHERE what::text = ANY (ARRAY['location_change'::character varying, 'poll'::character varying, 'midnight_location'::character varying]::text[])
"index_user_event_for_profile_views" btree (poi_id, created_at) WHERE what::text = 'poi_profile_shown'::text
"index_user_event_for_profile_views1" btree (poi_id, created_at, what) WHERE what::text = 'poi_profile_shown'::text
"index_user_event_for_profile_views2" btree (created_at, poi_id, what) WHERE what::text = 'poi_profile_shown'::text
"index_user_event_for_suggestion_notification" btree (((parameters ->> 'suggestion_id'::text)::integer), what) WHERE (parameters ->> 'suggestion_id'::text) IS NOT NULL
"index_user_event_for_visits" btree (poi_id, created_at, latitude, longitude) WHERE what::text = 'location_change'::text
"index_user_event_for_visits_3" btree (created_at, latitude, longitude, user_id) WHERE what::text = ANY (ARRAY['location_change'::character varying, 'poll'::character varying, 'midnight_location'::character varying]::text[])
"index_user_event_for_web_clicks" btree (poi_id, created_at) WHERE what::text = 'profile_weblink'::text
"index_user_event_on_created_at" btree (created_at)
"index_user_event_on_user_id_and_created_at_and_what" btree (user_id, created_at, what) WHERE what::text = ANY (ARRAY['location_change'::character varying, 'menu_tap'::character varying, 'poi_profile_shown'::character varying]::text[])
@ioquatix
Copy link
Author

explain analyze SELECT DISTINCT "user_event"."user_id" FROM "user_event" WHERE "user_event"."what" IN ('poll', 'location_change', 'midnight_location') AND ("user_event"."created_at" >= '2016-04-19 01:23:55') AND (latitude > -37.03079375089291 AND latitude < -36.67086424910709 AND longitude > 174.6307139779924 AND longitude < 175.0805140220076);
                                                                                                                           QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 HashAggregate  (cost=16677909.03..16678503.61 rows=59458 width=4) (actual time=540291.978..540336.233 rows=114316 loops=1)
   Group Key: user_id
   ->  Index Only Scan using index_user_event_for_visits_3 on user_event  (cost=0.70..16673302.32 rows=1842683 width=4) (actual time=0.716..524053.908 rows=17227730 loops=1)
         Index Cond: ((created_at >= '2016-04-19 01:23:55'::timestamp without time zone) AND (latitude > '-37.03079375089291'::numeric) AND (latitude < '-36.67086424910709'::numeric) AND (longitude > 174.6307139779924) AND (longitude < 175.0805140220076))
         Heap Fetches: 13225588
 Planning time: 0.556 ms
 Execution time: 540341.652 ms
(7 rows)

@ioquatix
Copy link
Author

After converting longitude/latitude to float4, and doing vacuum analyze user_event (restoring the index only scan):

geozone_data_production=> explain analyze SELECT DISTINCT "user_event"."user_id" FROM "user_event" WHERE "user_event"."what" IN ('poll', 'location_change', 'midnight_location') AND ("user_event"."created_at" >= '2016-04-19 01:23:55') AND (latitude > -37.03079375089291 AND latitude < -36.67086424910709 AND longitude > 174.6307139779924 AND longitude < 175.0805140220076);
                                                                                                                                                           QUERY PLAN

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------

HashAggregate  (cost=11934250.82..11934874.84 rows=62402 width=4) (actual time=42926.515..42981.003 rows=114997 loops=1)
   Group Key: user_id
   ->  Index Only Scan using index_user_event_for_visits_3 on user_event  (cost=0.57..11928524.90 rows=2290368 width=4) (actual time=0.171..36671.403 rows=18458352 loops=1)
         Index Cond: ((created_at >= '2016-04-19 01:23:55'::timestamp without time zone) AND (latitude > '-37.0307937508929115'::double precision) AND (latitude < '-36.6708642491070904'::double precision) AND (lo
ngitude > '174.6307139779924'::double precision) AND (longitude < '175.080514022007605'::double precision))
         Heap Fetches: 89770
 Planning time: 0.332 ms
 Execution time: 42987.434 ms
(7 rows)

@ioquatix
Copy link
Author

Here is the details of the table before:

geozone_data_production=> \d+
                                       List of relations
 Schema |                  Name                  |   Type   | Owner |    Size    | Description
--------+----------------------------------------+----------+-------+------------+-------------
 public | ar_internal_metadata                   | table    | http  | 16 kB      |
 public | poi                                    | table    | http  | 1966 MB    |
 public | poi_id_seq                             | sequence | http  | 8192 bytes |
 public | region                                 | table    | http  | 40 kB      |
 public | region_id_seq                          | sequence | http  | 8192 bytes |
 public | region_range                           | table    | http  | 120 kB     |
 public | region_range_id_seq                    | sequence | http  | 8192 bytes |
 public | schema_migrations                      | table    | http  | 16 kB      |
 public | suggestion                             | table    | http  | 48 kB      |
 public | suggestion_conversion                  | table    | http  | 2328 kB    |
 public | suggestion_conversion_aggregate        | table    | http  | 40 kB      |
 public | suggestion_conversion_aggregate_id_seq | sequence | http  | 8192 bytes |
 public | suggestion_conversion_id_seq           | sequence | http  | 8192 bytes |
 public | suggestion_id_seq                      | sequence | http  | 8192 bytes |
 public | user                                   | table    | http  | 101 MB     |
 public | user_event                             | table    | http  | 102 GB     |
 public | user_event_id_seq                      | sequence | http  | 8192 bytes |
 public | user_id_seq                            | sequence | http  | 8192 bytes |
(18 rows)

geozone_data_production=> \di+
                                                             List of relations
 Schema |                          Name                          | Type  | Owner |              Table              |  Size   | Description
--------+--------------------------------------------------------+-------+-------+---------------------------------+---------+-------------
 public | ar_internal_metadata_pkey                              | index | http  | ar_internal_metadata            | 16 kB   |
 public | index_region_range_on_region_id                        | index | http  | region_range                    | 64 kB   |
 public | index_suggestion_conversion_aggregate_on_suggestion_id | index | http  | suggestion_conversion_aggregate | 16 kB   |
 public | index_suggestion_conversion_on_suggestion_id           | index | http  | suggestion_conversion           | 1512 kB |
 public | index_suggestion_conversion_on_user_id                 | index | http  | suggestion_conversion           | 1488 kB |
 public | index_suggestion_on_poi_id                             | index | http  | suggestion                      | 16 kB   |
 public | index_suggestion_on_region_id                          | index | http  | suggestion                      | 16 kB   |
 public | index_user_event_for_calls                             | index | http  | user_event                      | 7312 kB |
 public | index_user_event_for_nearby_polls                      | index | http  | user_event                      | 21 GB   |
 public | index_user_event_for_profile_views                     | index | http  | user_event                      | 2405 MB |
 public | index_user_event_for_profile_views1                    | index | http  | user_event                      | 4449 MB |
 public | index_user_event_for_profile_views2                    | index | http  | user_event                      | 3665 MB |
 public | index_user_event_for_suggestion_notification           | index | http  | user_event                      | 8280 kB |
 public | index_user_event_for_visits_3                          | index | http  | user_event                      | 21 GB   |
 public | index_user_event_for_web_clicks                        | index | http  | user_event                      | 20 MB   |
 public | index_user_event_on_created_at                         | index | http  | user_event                      | 23 GB   |
 public | index_user_event_on_user_id_and_created_at_and_what    | index | http  | user_event                      | 20 GB   |
 public | poi_pkey                                               | index | http  | poi                             | 523 MB  |
 public | region_pkey                                            | index | http  | region                          | 16 kB   |
 public | region_range_pkey                                      | index | http  | region_range                    | 56 kB   |
 public | schema_migrations_pkey                                 | index | http  | schema_migrations               | 16 kB   |
 public | suggestion_conversion_aggregate_pkey                   | index | http  | suggestion_conversion_aggregate | 16 kB   |
 public | suggestion_conversion_pkey                             | index | http  | suggestion_conversion           | 1544 kB |
 public | suggestion_pkey                                        | index | http  | suggestion                      | 16 kB   |
 public | user_event_pkey                                        | index | http  | user_event                      | 18 GB   |
 public | user_pkey                                              | index | http  | user                            | 19 MB   |
(26 rows)

and after

geozone_data_production=> \d+
                                       List of relations
 Schema |                  Name                  |   Type   | Owner |    Size    | Description
--------+----------------------------------------+----------+-------+------------+-------------
 public | ar_internal_metadata                   | table    | http  | 16 kB      |
 public | poi                                    | table    | http  | 1966 MB    |
 public | poi_id_seq                             | sequence | http  | 8192 bytes |
 public | region                                 | table    | http  | 40 kB      |
 public | region_id_seq                          | sequence | http  | 8192 bytes |
 public | region_range                           | table    | http  | 120 kB     |
 public | region_range_id_seq                    | sequence | http  | 8192 bytes |
 public | schema_migrations                      | table    | http  | 16 kB      |
 public | suggestion                             | table    | http  | 48 kB      |
 public | suggestion_conversion                  | table    | http  | 2344 kB    |
 public | suggestion_conversion_aggregate        | table    | http  | 40 kB      |
 public | suggestion_conversion_aggregate_id_seq | sequence | http  | 8192 bytes |
 public | suggestion_conversion_id_seq           | sequence | http  | 8192 bytes |
 public | suggestion_id_seq                      | sequence | http  | 8192 bytes |
 public | user                                   | table    | http  | 101 MB     |
 public | user_event                             | table    | http  | 96 GB      |
 public | user_event_id_seq                      | sequence | http  | 8192 bytes |
 public | user_id_seq                            | sequence | http  | 8192 bytes |
(18 rows)

geozone_data_production=> \di+
                                                             List of relations
 Schema |                          Name                          | Type  | Owner |              Table              |  Size   | Description
--------+--------------------------------------------------------+-------+-------+---------------------------------+---------+-------------
 public | ar_internal_metadata_pkey                              | index | http  | ar_internal_metadata            | 16 kB   |
 public | index_region_range_on_region_id                        | index | http  | region_range                    | 64 kB   |
 public | index_suggestion_conversion_aggregate_on_suggestion_id | index | http  | suggestion_conversion_aggregate | 16 kB   |
 public | index_suggestion_conversion_on_suggestion_id           | index | http  | suggestion_conversion           | 1536 kB |
 public | index_suggestion_conversion_on_user_id                 | index | http  | suggestion_conversion           | 1488 kB |
 public | index_suggestion_on_poi_id                             | index | http  | suggestion                      | 16 kB   |
 public | index_suggestion_on_region_id                          | index | http  | suggestion                      | 16 kB   |
 public | index_user_event_for_calls                             | index | http  | user_event                      | 6888 kB |
 public | index_user_event_for_profile_views                     | index | http  | user_event                      | 2200 MB |
 public | index_user_event_for_suggestion_notification           | index | http  | user_event                      | 7936 kB |
 public | index_user_event_for_web_clicks                        | index | http  | user_event                      | 20 MB   |
 public | index_user_event_on_created_at                         | index | http  | user_event                      | 16 GB   |
 public | index_user_event_on_location_changes_user_id           | index | http  | user_event                      | 17 GB   |
 public | index_user_event_on_user_id_and_created_at_and_what    | index | http  | user_event                      | 19 GB   |
 public | poi_pkey                                               | index | http  | poi                             | 523 MB  |
 public | region_pkey                                            | index | http  | region                          | 16 kB   |
 public | region_range_pkey                                      | index | http  | region_range                    | 56 kB   |
 public | schema_migrations_pkey                                 | index | http  | schema_migrations               | 16 kB   |
 public | suggestion_conversion_aggregate_pkey                   | index | http  | suggestion_conversion_aggregate | 16 kB   |
 public | suggestion_conversion_pkey                             | index | http  | suggestion_conversion           | 1552 kB |
 public | suggestion_pkey                                        | index | http  | suggestion                      | 16 kB   |
 public | user_event_pkey                                        | index | http  | user_event                      | 16 GB   |
 public | user_pkey                                              | index | http  | user                            | 19 MB   |
(23 rows)

@ioquatix
Copy link
Author

After doing the alter table, it wouldn't do the index only scan. Found out you need to vacuum analyze user_event from http://ask.use-the-index-luke.com/questions/148/why-is-this-postgres-query-doing-a-bitmap-heap-scan-after-the-index-scan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment