-
-
Save psalaberria002/e91956d94e82d706d65da7111a40ca24 to your computer and use it in GitHub Desktop.
18-05-07 20:21:10,703 INFO 76:cognite_logger.py(1) - Live worker enabled. Initializing queue... | |
2018-05-07 20:21:10,710 INFO 76:cognite_logger.py(1) - Backfill worker enabled. Initializing queue... | |
2018-05-07 20:21:10,715 INFO 76:cognite_logger.py(1) - ElasticsearchLoader initialized, not started yet. | |
2018-05-07 20:21:10,717 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
2018-05-07 20:21:10,719 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
2018-05-07 20:21:10,719 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:10,720 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.173182 seconds. | |
2018-05-07 20:21:10,720 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
2018-05-07 20:21:10,722 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/resource-changes-livewith subscription projects/cognitedata-production/subscriptions/search-loader-resource-changes-live | |
2018-05-07 20:21:10,722 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:21:10,723 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
2018-05-07 20:21:10,723 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
2018-05-07 20:21:10,724 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:10,724 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.205845 seconds. | |
2018-05-07 20:21:10,724 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
2018-05-07 20:21:10,725 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/search-loader-backfillwith subscription projects/cognitedata-production/subscriptions/search-loader-backfill | |
2018-05-07 20:21:10,725 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:21:10,829 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
2018-05-07 20:21:10,920 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
2018-05-07 20:21:10,924 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
2018-05-07 20:21:10,926 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
2018-05-07 20:21:11,052 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
2018-05-07 20:21:11,054 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
2018-05-07 20:21:14,896 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:14,897 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.357022 seconds. | |
2018-05-07 20:21:17,256 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:17,256 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.351780 seconds. | |
2018-05-07 20:21:18,938 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:18,939 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.336647 seconds. | |
2018-05-07 20:21:19,611 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:19,611 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.521509 seconds. | |
2018-05-07 20:21:21,278 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:21,278 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.942409 seconds. | |
2018-05-07 20:21:22,135 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:22,136 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.968433 seconds. | |
2018-05-07 20:21:28,110 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:28,111 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.788305 seconds. | |
2018-05-07 20:21:28,222 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:28,222 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.065991 seconds. | |
2018-05-07 20:21:29,289 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:29,289 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.907441 seconds. | |
2018-05-07 20:21:31,903 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:31,903 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.050113 seconds. | |
2018-05-07 20:21:31,954 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:31,954 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.964171 seconds. | |
2018-05-07 20:21:34,202 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:34,202 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.203038 seconds. | |
2018-05-07 20:21:36,923 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:36,923 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.861826 seconds. | |
2018-05-07 20:21:38,787 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:38,788 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.311823 seconds. | |
2018-05-07 20:21:40,412 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:40,412 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.334087 seconds. | |
2018-05-07 20:21:46,752 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:46,752 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.509286 seconds. | |
2018-05-07 20:21:47,108 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:47,108 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.650154 seconds. | |
2018-05-07 20:21:50,128 DEBUG 440:bidi.py(1) - recved response. | |
2018-05-07 20:21:50,128 DEBUG 340:streaming_pull_manager.py(1) - Scheduling callbacks for 1 messages. | |
2018-05-07 20:21:50,129 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
2018-05-07 20:21:50,130 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:21:50,137 DEBUG 71:cognite_logger.py(1) - b'345' | |
2018-05-07 20:21:50,141 ERROR 86:cognite_logger.py(1) - Unhandled exception. The message will be unacknowledged after the deadline. 'int' object has no attribute 'get' | |
Traceback (most recent call last): | |
File "/cognite_search_loader/loader.py", line 130, in process_live_message | |
resource_type = data.get("resourceType").lower() | |
AttributeError: 'int' object has no attribute 'get' | |
2018-05-07 20:21:51,763 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:51,764 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:21:51,764 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.069842 seconds. | |
2018-05-07 20:21:53,266 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:53,266 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.946386 seconds. | |
2018-05-07 20:21:54,837 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:54,838 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:21:54,891 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.920633 seconds. | |
2018-05-07 20:21:55,813 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:55,813 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:21:55,813 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.307516 seconds. | |
2018-05-07 20:21:57,122 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:57,122 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:21:57,122 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.342947 seconds. | |
2018-05-07 20:21:59,218 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:21:59,219 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.629198 seconds. | |
2018-05-07 20:22:01,470 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:01,470 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:01,470 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.400186 seconds. | |
2018-05-07 20:22:05,872 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:05,872 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:05,872 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.418915 seconds. | |
2018-05-07 20:22:06,292 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:06,292 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:06,292 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.967860 seconds. | |
2018-05-07 20:22:06,856 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:06,856 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.723435 seconds. | |
2018-05-07 20:22:13,262 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:13,262 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:13,262 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.711005 seconds. | |
2018-05-07 20:22:15,588 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:15,589 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.819970 seconds. | |
2018-05-07 20:22:19,412 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:19,413 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.137266 seconds. | |
2018-05-07 20:22:19,976 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:19,977 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:19,977 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.700402 seconds. | |
2018-05-07 20:22:26,684 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:26,684 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:26,685 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.327550 seconds. | |
2018-05-07 20:22:27,556 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:27,556 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.722393 seconds. | |
2018-05-07 20:22:35,021 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:35,022 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:35,022 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.204804 seconds. | |
2018-05-07 20:22:35,281 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:35,282 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.481293 seconds. | |
2018-05-07 20:22:39,768 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:39,768 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.338330 seconds. | |
2018-05-07 20:22:42,234 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:42,234 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:42,234 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.404186 seconds. | |
2018-05-07 20:22:42,639 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:42,640 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:42,640 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.626212 seconds. | |
2018-05-07 20:22:44,111 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:44,111 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.242587 seconds. | |
2018-05-07 20:22:44,354 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:44,354 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.094260 seconds. | |
2018-05-07 20:22:49,452 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:49,452 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.605809 seconds. | |
2018-05-07 20:22:50,059 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:50,060 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.865825 seconds. | |
2018-05-07 20:22:50,267 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:50,267 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:50,268 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.597384 seconds. | |
2018-05-07 20:22:51,867 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:51,867 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:51,868 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.357316 seconds. | |
2018-05-07 20:22:57,095 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
2018-05-07 20:22:57,934 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:57,934 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.477696 seconds. | |
2018-05-07 20:22:59,232 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:22:59,232 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:22:59,232 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.081595 seconds. | |
2018-05-07 20:23:01,316 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:01,316 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:01,316 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.302564 seconds. | |
2018-05-07 20:23:02,416 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:02,417 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.300140 seconds. | |
2018-05-07 20:23:04,623 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:04,623 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:04,623 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.041896 seconds. | |
2018-05-07 20:23:08,668 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:08,669 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:08,669 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.373342 seconds. | |
2018-05-07 20:23:08,723 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:08,724 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.928441 seconds. | |
2018-05-07 20:23:10,044 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:10,044 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:10,044 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.681042 seconds. | |
2018-05-07 20:23:13,729 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:13,730 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:13,730 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.482348 seconds. | |
2018-05-07 20:23:16,660 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:16,660 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.521659 seconds. | |
2018-05-07 20:23:22,221 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:22,221 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:22,221 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.541945 seconds. | |
2018-05-07 20:23:23,189 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:23,189 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.401518 seconds. | |
2018-05-07 20:23:26,594 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:26,595 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.721336 seconds. | |
2018-05-07 20:23:26,767 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:26,768 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:26,768 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.357067 seconds. | |
2018-05-07 20:23:33,132 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:33,132 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:33,132 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.250864 seconds. | |
2018-05-07 20:23:33,318 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:33,318 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.131657 seconds. | |
2018-05-07 20:23:37,387 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:37,388 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
2018-05-07 20:23:37,388 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.474147 seconds. | |
2018-05-07 20:23:38,455 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:38,455 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.824235 seconds. | |
2018-05-07 20:23:40,281 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:40,282 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.290993 seconds. | |
^C | |
~ ⎈ gke-cognitedata-production | |
$ k logs -f search-loader-5c8fd5bd57-x9xnk | |
2018-05-07 20:23:35,025 INFO 76:cognite_logger.py(1) - Live worker enabled. Initializing queue... | |
2018-05-07 20:23:35,033 INFO 76:cognite_logger.py(1) - Backfill worker enabled. Initializing queue... | |
2018-05-07 20:23:35,039 INFO 76:cognite_logger.py(1) - ElasticsearchLoader initialized, not started yet. | |
2018-05-07 20:23:35,042 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
2018-05-07 20:23:35,093 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
2018-05-07 20:23:35,093 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:35,094 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
2018-05-07 20:23:35,095 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.730845 seconds. | |
2018-05-07 20:23:35,096 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:23:35,096 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/resource-changes-livewith subscription projects/cognitedata-production/subscriptions/search-loader-resource-changes-live | |
2018-05-07 20:23:35,097 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
2018-05-07 20:23:35,098 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
2018-05-07 20:23:35,098 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:35,098 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
2018-05-07 20:23:35,099 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/search-loader-backfillwith subscription projects/cognitedata-production/subscriptions/search-loader-backfill | |
2018-05-07 20:23:35,098 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.322453 seconds. | |
2018-05-07 20:23:35,099 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:23:35,205 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
2018-05-07 20:23:35,297 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
2018-05-07 20:23:35,299 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
2018-05-07 20:23:35,299 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
2018-05-07 20:23:35,397 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
2018-05-07 20:23:35,402 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
2018-05-07 20:23:38,424 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:38,424 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.208393 seconds. | |
2018-05-07 20:23:42,835 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:42,835 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.763262 seconds. | |
2018-05-07 20:23:46,641 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:46,641 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.059000 seconds. | |
2018-05-07 20:23:50,606 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:50,606 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.936351 seconds. | |
2018-05-07 20:23:50,705 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:50,705 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.493366 seconds. | |
2018-05-07 20:23:51,543 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:51,544 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.370397 seconds. | |
2018-05-07 20:23:55,963 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:55,964 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.451475 seconds. | |
2018-05-07 20:23:57,417 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:57,418 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.904849 seconds. | |
2018-05-07 20:23:58,206 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:58,206 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.443527 seconds. | |
2018-05-07 20:23:58,651 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:58,651 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.535724 seconds. | |
2018-05-07 20:24:02,191 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:02,191 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.092101 seconds. | |
2018-05-07 20:24:02,324 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:02,324 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.445057 seconds. | |
2018-05-07 20:24:03,771 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:03,771 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.937518 seconds. | |
2018-05-07 20:24:07,288 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:07,289 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.989881 seconds. | |
2018-05-07 20:24:07,713 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:07,713 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.826623 seconds. | |
2018-05-07 20:24:09,542 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:09,542 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.622086 seconds. | |
2018-05-07 20:24:10,165 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:10,165 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.280877 seconds. | |
2018-05-07 20:24:11,282 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:11,282 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.222935 seconds. | |
2018-05-07 20:24:14,448 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:14,449 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.875739 seconds. | |
2018-05-07 20:24:16,511 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:16,511 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.144540 seconds. | |
2018-05-07 20:24:21,661 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:21,661 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.476803 seconds. | |
2018-05-07 20:24:23,325 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:23,326 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.667869 seconds. | |
2018-05-07 20:24:23,994 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:23,995 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.218226 seconds. | |
2018-05-07 20:24:26,143 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:26,143 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.536331 seconds. | |
2018-05-07 20:24:26,215 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:26,215 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.604584 seconds. | |
2018-05-07 20:24:26,821 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:26,821 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.705008 seconds. | |
2018-05-07 20:24:27,681 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:27,682 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.322410 seconds. | |
2018-05-07 20:24:31,531 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:31,532 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.477748 seconds. | |
2018-05-07 20:24:34,011 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:34,011 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.743311 seconds. | |
2018-05-07 20:24:35,014 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:35,014 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.153356 seconds. | |
2018-05-07 20:24:41,173 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:41,174 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.699673 seconds. | |
2018-05-07 20:24:41,874 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:41,875 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.206370 seconds. | |
2018-05-07 20:24:42,763 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:42,764 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.274700 seconds. | |
2018-05-07 20:24:45,041 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:45,041 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.749096 seconds. | |
2018-05-07 20:24:46,082 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:46,082 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.384495 seconds. | |
2018-05-07 20:24:48,248 DEBUG 440:bidi.py(1) - recved response. | |
2018-05-07 20:24:48,248 DEBUG 340:streaming_pull_manager.py(1) - Scheduling callbacks for 1 messages. | |
2018-05-07 20:24:48,249 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
2018-05-07 20:24:48,293 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:24:48,305 DEBUG 71:cognite_logger.py(1) - b'ghjkl' | |
2018-05-07 20:24:48,311 ERROR 86:cognite_logger.py(1) - Could not parse message, getting rid of it. Expecting value: line 1 column 1 (char 0) | |
Traceback (most recent call last): | |
File "/cognite_search_loader/loader.py", line 120, in process_live_message | |
data = json.loads(message.data) | |
File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads | |
return _default_decoder.decode(s) | |
File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode | |
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) | |
File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode | |
raise JSONDecodeError("Expecting value", s, err.value) from None | |
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) | |
2018-05-07 20:24:48,312 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
2018-05-07 20:24:48,793 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:48,794 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.272399 seconds. | |
2018-05-07 20:24:52,070 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:52,070 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.218485 seconds. | |
2018-05-07 20:24:54,471 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:54,472 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.696924 seconds. | |
2018-05-07 20:24:55,292 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:55,292 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.860597 seconds. | |
2018-05-07 20:24:55,431 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
2018-05-07 20:25:01,175 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:01,175 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.276403 seconds. | |
2018-05-07 20:25:02,155 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:02,155 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.027589 seconds. | |
2018-05-07 20:25:03,454 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:03,454 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.777448 seconds. | |
2018-05-07 20:25:07,185 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:07,185 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.296408 seconds. | |
2018-05-07 20:25:08,233 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:08,233 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.980049 seconds. | |
2018-05-07 20:25:12,483 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:12,484 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.068263 seconds. | |
2018-05-07 20:25:14,554 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:14,554 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.770624 seconds. | |
2018-05-07 20:25:17,223 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:17,223 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.750148 seconds. | |
2018-05-07 20:25:20,977 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:20,978 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.130011 seconds. | |
2018-05-07 20:25:21,327 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:21,327 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.685115 seconds. | |
2018-05-07 20:25:22,013 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:22,014 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.288631 seconds. | |
2018-05-07 20:25:24,303 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:24,304 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.271615 seconds. | |
2018-05-07 20:25:25,112 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:25,112 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.278409 seconds. | |
2018-05-07 20:25:27,579 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:27,579 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.451800 seconds. | |
2018-05-07 20:25:29,394 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:29,394 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.617179 seconds. | |
2018-05-07 20:25:34,016 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:34,017 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.046940 seconds. | |
2018-05-07 20:25:34,037 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:34,037 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.474646 seconds. | |
2018-05-07 20:25:37,066 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:37,066 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.801040 seconds. | |
2018-05-07 20:25:38,514 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:38,514 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.342559 seconds. | |
2018-05-07 20:25:38,857 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:38,857 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.864900 seconds. | |
2018-05-07 20:25:45,873 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:45,874 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.157340 seconds. | |
2018-05-07 20:25:46,724 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:46,724 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.036529 seconds. | |
2018-05-07 20:25:47,033 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:47,033 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.626737 seconds. | |
^C | |
~ ⎈ gke-cognitedata-production | |
$ k logs -f search-loader-5c8fd5bd57-x9xnk | |
2018-05-07 20:23:35,025 INFO 76:cognite_logger.py(1) - Live worker enabled. Initializing queue... | |
2018-05-07 20:23:35,033 INFO 76:cognite_logger.py(1) - Backfill worker enabled. Initializing queue... | |
2018-05-07 20:23:35,039 INFO 76:cognite_logger.py(1) - ElasticsearchLoader initialized, not started yet. | |
2018-05-07 20:23:35,042 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
2018-05-07 20:23:35,093 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
2018-05-07 20:23:35,093 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:35,094 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
2018-05-07 20:23:35,095 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.730845 seconds. | |
2018-05-07 20:23:35,096 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:23:35,096 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/resource-changes-livewith subscription projects/cognitedata-production/subscriptions/search-loader-resource-changes-live | |
2018-05-07 20:23:35,097 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
2018-05-07 20:23:35,098 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
2018-05-07 20:23:35,098 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:35,098 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
2018-05-07 20:23:35,099 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/search-loader-backfillwith subscription projects/cognitedata-production/subscriptions/search-loader-backfill | |
2018-05-07 20:23:35,098 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.322453 seconds. | |
2018-05-07 20:23:35,099 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:23:35,205 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
2018-05-07 20:23:35,297 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
2018-05-07 20:23:35,299 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
2018-05-07 20:23:35,299 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
2018-05-07 20:23:35,397 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
2018-05-07 20:23:35,402 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
2018-05-07 20:23:38,424 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:38,424 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.208393 seconds. | |
2018-05-07 20:23:42,835 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:42,835 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.763262 seconds. | |
2018-05-07 20:23:46,641 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:46,641 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.059000 seconds. | |
2018-05-07 20:23:50,606 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:50,606 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.936351 seconds. | |
2018-05-07 20:23:50,705 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:50,705 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.493366 seconds. | |
2018-05-07 20:23:51,543 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:51,544 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.370397 seconds. | |
2018-05-07 20:23:55,963 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:55,964 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.451475 seconds. | |
2018-05-07 20:23:57,417 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:57,418 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.904849 seconds. | |
2018-05-07 20:23:58,206 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:58,206 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.443527 seconds. | |
2018-05-07 20:23:58,651 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:23:58,651 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.535724 seconds. | |
2018-05-07 20:24:02,191 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:02,191 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.092101 seconds. | |
2018-05-07 20:24:02,324 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:02,324 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.445057 seconds. | |
2018-05-07 20:24:03,771 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:03,771 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.937518 seconds. | |
2018-05-07 20:24:07,288 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:07,289 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.989881 seconds. | |
2018-05-07 20:24:07,713 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:07,713 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.826623 seconds. | |
2018-05-07 20:24:09,542 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:09,542 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.622086 seconds. | |
2018-05-07 20:24:10,165 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:10,165 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.280877 seconds. | |
2018-05-07 20:24:11,282 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:11,282 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.222935 seconds. | |
2018-05-07 20:24:14,448 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:14,449 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.875739 seconds. | |
2018-05-07 20:24:16,511 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:16,511 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.144540 seconds. | |
2018-05-07 20:24:21,661 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:21,661 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.476803 seconds. | |
2018-05-07 20:24:23,325 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:23,326 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.667869 seconds. | |
2018-05-07 20:24:23,994 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:23,995 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.218226 seconds. | |
2018-05-07 20:24:26,143 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:26,143 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.536331 seconds. | |
2018-05-07 20:24:26,215 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:26,215 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.604584 seconds. | |
2018-05-07 20:24:26,821 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:26,821 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.705008 seconds. | |
2018-05-07 20:24:27,681 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:27,682 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.322410 seconds. | |
2018-05-07 20:24:31,531 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:31,532 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.477748 seconds. | |
2018-05-07 20:24:34,011 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:34,011 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.743311 seconds. | |
2018-05-07 20:24:35,014 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:35,014 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.153356 seconds. | |
2018-05-07 20:24:41,173 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:41,174 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.699673 seconds. | |
2018-05-07 20:24:41,874 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:41,875 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.206370 seconds. | |
2018-05-07 20:24:42,763 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:42,764 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.274700 seconds. | |
2018-05-07 20:24:45,041 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:45,041 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.749096 seconds. | |
2018-05-07 20:24:46,082 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:46,082 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.384495 seconds. | |
2018-05-07 20:24:48,248 DEBUG 440:bidi.py(1) - recved response. | |
2018-05-07 20:24:48,248 DEBUG 340:streaming_pull_manager.py(1) - Scheduling callbacks for 1 messages. | |
2018-05-07 20:24:48,249 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
2018-05-07 20:24:48,293 DEBUG 438:bidi.py(1) - waiting for recv. | |
2018-05-07 20:24:48,305 DEBUG 71:cognite_logger.py(1) - b'ghjkl' | |
2018-05-07 20:24:48,311 ERROR 86:cognite_logger.py(1) - Could not parse message, getting rid of it. Expecting value: line 1 column 1 (char 0) | |
Traceback (most recent call last): | |
File "/cognite_search_loader/loader.py", line 120, in process_live_message | |
data = json.loads(message.data) | |
File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads | |
return _default_decoder.decode(s) | |
File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode | |
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) | |
File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode | |
raise JSONDecodeError("Expecting value", s, err.value) from None | |
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) | |
2018-05-07 20:24:48,312 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
2018-05-07 20:24:48,793 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:48,794 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.272399 seconds. | |
2018-05-07 20:24:52,070 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:52,070 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.218485 seconds. | |
2018-05-07 20:24:54,471 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:54,472 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.696924 seconds. | |
2018-05-07 20:24:55,292 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:24:55,292 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.860597 seconds. | |
2018-05-07 20:24:55,431 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
2018-05-07 20:25:01,175 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:01,175 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.276403 seconds. | |
2018-05-07 20:25:02,155 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:02,155 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.027589 seconds. | |
2018-05-07 20:25:03,454 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:03,454 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.777448 seconds. | |
2018-05-07 20:25:07,185 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:07,185 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.296408 seconds. | |
2018-05-07 20:25:08,233 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:08,233 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.980049 seconds. | |
2018-05-07 20:25:12,483 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:12,484 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.068263 seconds. | |
2018-05-07 20:25:14,554 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:14,554 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.770624 seconds. | |
2018-05-07 20:25:17,223 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:17,223 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.750148 seconds. | |
2018-05-07 20:25:20,977 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:20,978 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.130011 seconds. | |
2018-05-07 20:25:21,327 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:21,327 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.685115 seconds. | |
2018-05-07 20:25:22,013 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:22,014 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.288631 seconds. | |
2018-05-07 20:25:24,303 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:24,304 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.271615 seconds. | |
2018-05-07 20:25:25,112 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:25,112 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.278409 seconds. | |
2018-05-07 20:25:27,579 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:27,579 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.451800 seconds. | |
2018-05-07 20:25:29,394 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:29,394 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.617179 seconds. | |
2018-05-07 20:25:34,016 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:34,017 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.046940 seconds. | |
2018-05-07 20:25:34,037 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:34,037 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.474646 seconds. | |
2018-05-07 20:25:37,066 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:37,066 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.801040 seconds. | |
2018-05-07 20:25:38,514 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:38,514 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.342559 seconds. | |
2018-05-07 20:25:38,857 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:38,857 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.864900 seconds. | |
2018-05-07 20:25:45,873 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:45,874 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.157340 seconds. | |
2018-05-07 20:25:46,724 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:46,724 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.036529 seconds. | |
2018-05-07 20:25:47,033 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:47,033 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.626737 seconds. | |
2018-05-07 20:25:52,766 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:52,766 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.108176 seconds. | |
2018-05-07 20:25:52,875 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:52,875 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.432782 seconds. | |
2018-05-07 20:25:55,663 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:55,664 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.603902 seconds. | |
2018-05-07 20:25:56,310 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:25:56,311 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.093734 seconds. | |
2018-05-07 20:25:57,377 DEBUG 448:bidi.py(1) - Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code. | |
Traceback (most recent call last): | |
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next | |
return six.next(self._wrapped) | |
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__ | |
return self._next() | |
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next | |
raise self | |
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])> | |
The above exception was the direct cause of the following exception: | |
Traceback (most recent call last): | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 348, in _recoverable | |
return method(*args, **kwargs) | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv | |
return next(self.call) | |
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next | |
six.raise_from(exceptions.from_grpc_error(exc), exc) | |
File "<string>", line 3, in raise_from | |
google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75] | |
During handling of the above exception, another exception occurred: | |
Traceback (most recent call last): | |
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next | |
return six.next(self._wrapped) | |
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__ | |
return self._next() | |
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next | |
raise self | |
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])> | |
The above exception was the direct cause of the following exception: | |
Traceback (most recent call last): | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 439, in _thread_main | |
response = self._bidi_rpc.recv() | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 365, in recv | |
super(ResumableBidiRpc, self).recv) | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 357, in _recoverable | |
return method(*args, **kwargs) | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv | |
return next(self.call) | |
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next | |
six.raise_from(exceptions.from_grpc_error(exc), exc) | |
File "<string>", line 3, in raise_from | |
google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75] | |
2018-05-07 20:25:57,378 INFO 455:bidi.py(1) - Thread-ConsumeBidirectionalStream exiting | |
2018-05-07 20:25:58,108 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
2018-05-07 20:25:58,407 INFO 164:leaser.py(1) - Thread-LeaseMaintainer exiting. | |
2018-05-07 20:26:02,275 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:02,275 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.549899 seconds. | |
2018-05-07 20:26:09,833 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:09,833 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.363373 seconds. | |
2018-05-07 20:26:10,197 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:10,198 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.908064 seconds. | |
2018-05-07 20:26:14,110 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:14,110 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.179245 seconds. | |
2018-05-07 20:26:21,296 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:21,296 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.171606 seconds. | |
2018-05-07 20:26:25,398 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
2018-05-07 20:26:26,473 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:26,474 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.733286 seconds. | |
2018-05-07 20:26:32,213 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:32,214 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.333760 seconds. | |
2018-05-07 20:26:38,552 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:38,552 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.954605 seconds. | |
2018-05-07 20:26:45,514 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:45,514 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.039095 seconds. | |
2018-05-07 20:26:53,555 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:53,555 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.136897 seconds. | |
2018-05-07 20:26:53,693 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:26:53,693 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.990966 seconds. | |
2018-05-07 20:27:02,689 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:02,689 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.007966 seconds. | |
2018-05-07 20:27:06,702 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:06,702 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.666940 seconds. | |
2018-05-07 20:27:14,373 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:14,374 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.576805 seconds. | |
2018-05-07 20:27:21,958 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:21,959 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.749982 seconds. | |
2018-05-07 20:27:27,715 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:27,715 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.272717 seconds. | |
2018-05-07 20:27:30,991 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:30,991 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.771143 seconds. | |
2018-05-07 20:27:33,763 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:33,763 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.827710 seconds. | |
2018-05-07 20:27:34,592 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:34,592 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.229114 seconds. | |
2018-05-07 20:27:34,822 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:34,822 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.949239 seconds. | |
2018-05-07 20:27:43,773 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:43,774 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.192543 seconds. | |
2018-05-07 20:27:50,971 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:50,971 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.647281 seconds. | |
2018-05-07 20:27:59,623 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:27:59,624 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.893704 seconds. | |
2018-05-07 20:28:03,522 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:28:03,522 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.280092 seconds. | |
2018-05-07 20:28:10,810 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:28:10,810 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.781080 seconds. | |
2018-05-07 20:28:18,594 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
2018-05-07 20:28:18,595 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.834090 seconds. | |
2018-05-07 20:28:19,308 DEBUG 448:bidi.py(1) - Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code. | |
Traceback (most recent call last): | |
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next | |
return six.next(self._wrapped) | |
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__ | |
return self._next() | |
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next | |
raise self | |
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])> | |
The above exception was the direct cause of the following exception: | |
Traceback (most recent call last): | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 348, in _recoverable | |
return method(*args, **kwargs) | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv | |
return next(self.call) | |
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next | |
six.raise_from(exceptions.from_grpc_error(exc), exc) | |
File "<string>", line 3, in raise_from | |
google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75] | |
During handling of the above exception, another exception occurred: | |
Traceback (most recent call last): | |
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next | |
return six.next(self._wrapped) | |
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__ | |
return self._next() | |
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next | |
raise self | |
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])> | |
The above exception was the direct cause of the following exception: | |
Traceback (most recent call last): | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 439, in _thread_main | |
response = self._bidi_rpc.recv() | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 365, in recv | |
super(ResumableBidiRpc, self).recv) | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 357, in _recoverable | |
return method(*args, **kwargs) | |
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv | |
return next(self.call) | |
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next | |
six.raise_from(exceptions.from_grpc_error(exc), exc) | |
File "<string>", line 3, in raise_from | |
google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75] | |
2018-05-07 20:28:19,309 INFO 455:bidi.py(1) - Thread-ConsumeBidirectionalStream exiting | |
2018-05-07 20:28:19,429 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. |
hi,
What's the fix for the above error?
I am also getting a similar error, when there is no message in PubSub:
code:
while True:
response = subscriber.api.pull(sub_path, max_messages=int(_NUM_MESSAGES))
---------
sleep(0.1)
Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 549, in call
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
that was a long time ago, and I think it was fixed by upgrading to newer versions of the pubsub library. We are not using the Python sdk anymore, can't help.
Another error