Created
May 7, 2018 20:46
-
-
Save psalaberria002/e91956d94e82d706d65da7111a40ca24 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 18-05-07 20:21:10,703 INFO 76:cognite_logger.py(1) - Live worker enabled. Initializing queue... | |
| 2018-05-07 20:21:10,710 INFO 76:cognite_logger.py(1) - Backfill worker enabled. Initializing queue... | |
| 2018-05-07 20:21:10,715 INFO 76:cognite_logger.py(1) - ElasticsearchLoader initialized, not started yet. | |
| 2018-05-07 20:21:10,717 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
| 2018-05-07 20:21:10,719 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
| 2018-05-07 20:21:10,719 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:10,720 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.173182 seconds. | |
| 2018-05-07 20:21:10,720 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
| 2018-05-07 20:21:10,722 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/resource-changes-livewith subscription projects/cognitedata-production/subscriptions/search-loader-resource-changes-live | |
| 2018-05-07 20:21:10,722 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:21:10,723 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
| 2018-05-07 20:21:10,723 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
| 2018-05-07 20:21:10,724 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:10,724 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.205845 seconds. | |
| 2018-05-07 20:21:10,724 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
| 2018-05-07 20:21:10,725 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/search-loader-backfillwith subscription projects/cognitedata-production/subscriptions/search-loader-backfill | |
| 2018-05-07 20:21:10,725 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:21:10,829 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
| 2018-05-07 20:21:10,920 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
| 2018-05-07 20:21:10,924 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
| 2018-05-07 20:21:10,926 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
| 2018-05-07 20:21:11,052 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
| 2018-05-07 20:21:11,054 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
| 2018-05-07 20:21:14,896 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:14,897 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.357022 seconds. | |
| 2018-05-07 20:21:17,256 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:17,256 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.351780 seconds. | |
| 2018-05-07 20:21:18,938 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:18,939 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.336647 seconds. | |
| 2018-05-07 20:21:19,611 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:19,611 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.521509 seconds. | |
| 2018-05-07 20:21:21,278 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:21,278 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.942409 seconds. | |
| 2018-05-07 20:21:22,135 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:22,136 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.968433 seconds. | |
| 2018-05-07 20:21:28,110 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:28,111 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.788305 seconds. | |
| 2018-05-07 20:21:28,222 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:28,222 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.065991 seconds. | |
| 2018-05-07 20:21:29,289 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:29,289 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.907441 seconds. | |
| 2018-05-07 20:21:31,903 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:31,903 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.050113 seconds. | |
| 2018-05-07 20:21:31,954 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:31,954 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.964171 seconds. | |
| 2018-05-07 20:21:34,202 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:34,202 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.203038 seconds. | |
| 2018-05-07 20:21:36,923 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:36,923 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.861826 seconds. | |
| 2018-05-07 20:21:38,787 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:38,788 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.311823 seconds. | |
| 2018-05-07 20:21:40,412 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:40,412 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.334087 seconds. | |
| 2018-05-07 20:21:46,752 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:46,752 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.509286 seconds. | |
| 2018-05-07 20:21:47,108 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:47,108 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.650154 seconds. | |
| 2018-05-07 20:21:50,128 DEBUG 440:bidi.py(1) - recved response. | |
| 2018-05-07 20:21:50,128 DEBUG 340:streaming_pull_manager.py(1) - Scheduling callbacks for 1 messages. | |
| 2018-05-07 20:21:50,129 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
| 2018-05-07 20:21:50,130 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:21:50,137 DEBUG 71:cognite_logger.py(1) - b'345' | |
| 2018-05-07 20:21:50,141 ERROR 86:cognite_logger.py(1) - Unhandled exception. The message will be unacknowledged after the deadline. 'int' object has no attribute 'get' | |
| Traceback (most recent call last): | |
| File "/cognite_search_loader/loader.py", line 130, in process_live_message | |
| resource_type = data.get("resourceType").lower() | |
| AttributeError: 'int' object has no attribute 'get' | |
| 2018-05-07 20:21:51,763 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:51,764 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:21:51,764 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.069842 seconds. | |
| 2018-05-07 20:21:53,266 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:53,266 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.946386 seconds. | |
| 2018-05-07 20:21:54,837 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:54,838 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:21:54,891 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.920633 seconds. | |
| 2018-05-07 20:21:55,813 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:55,813 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:21:55,813 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.307516 seconds. | |
| 2018-05-07 20:21:57,122 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:57,122 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:21:57,122 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.342947 seconds. | |
| 2018-05-07 20:21:59,218 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:21:59,219 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.629198 seconds. | |
| 2018-05-07 20:22:01,470 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:01,470 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:01,470 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.400186 seconds. | |
| 2018-05-07 20:22:05,872 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:05,872 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:05,872 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.418915 seconds. | |
| 2018-05-07 20:22:06,292 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:06,292 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:06,292 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.967860 seconds. | |
| 2018-05-07 20:22:06,856 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:06,856 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.723435 seconds. | |
| 2018-05-07 20:22:13,262 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:13,262 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:13,262 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.711005 seconds. | |
| 2018-05-07 20:22:15,588 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:15,589 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.819970 seconds. | |
| 2018-05-07 20:22:19,412 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:19,413 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.137266 seconds. | |
| 2018-05-07 20:22:19,976 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:19,977 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:19,977 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.700402 seconds. | |
| 2018-05-07 20:22:26,684 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:26,684 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:26,685 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.327550 seconds. | |
| 2018-05-07 20:22:27,556 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:27,556 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.722393 seconds. | |
| 2018-05-07 20:22:35,021 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:35,022 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:35,022 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.204804 seconds. | |
| 2018-05-07 20:22:35,281 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:35,282 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.481293 seconds. | |
| 2018-05-07 20:22:39,768 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:39,768 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.338330 seconds. | |
| 2018-05-07 20:22:42,234 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:42,234 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:42,234 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.404186 seconds. | |
| 2018-05-07 20:22:42,639 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:42,640 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:42,640 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.626212 seconds. | |
| 2018-05-07 20:22:44,111 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:44,111 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.242587 seconds. | |
| 2018-05-07 20:22:44,354 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:44,354 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.094260 seconds. | |
| 2018-05-07 20:22:49,452 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:49,452 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.605809 seconds. | |
| 2018-05-07 20:22:50,059 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:50,060 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.865825 seconds. | |
| 2018-05-07 20:22:50,267 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:50,267 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:50,268 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.597384 seconds. | |
| 2018-05-07 20:22:51,867 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:51,867 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:51,868 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.357316 seconds. | |
| 2018-05-07 20:22:57,095 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
| 2018-05-07 20:22:57,934 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:57,934 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.477696 seconds. | |
| 2018-05-07 20:22:59,232 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:22:59,232 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:22:59,232 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.081595 seconds. | |
| 2018-05-07 20:23:01,316 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:01,316 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:01,316 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.302564 seconds. | |
| 2018-05-07 20:23:02,416 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:02,417 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.300140 seconds. | |
| 2018-05-07 20:23:04,623 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:04,623 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:04,623 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.041896 seconds. | |
| 2018-05-07 20:23:08,668 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:08,669 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:08,669 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.373342 seconds. | |
| 2018-05-07 20:23:08,723 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:08,724 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.928441 seconds. | |
| 2018-05-07 20:23:10,044 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:10,044 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:10,044 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.681042 seconds. | |
| 2018-05-07 20:23:13,729 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:13,730 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:13,730 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.482348 seconds. | |
| 2018-05-07 20:23:16,660 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:16,660 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.521659 seconds. | |
| 2018-05-07 20:23:22,221 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:22,221 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:22,221 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.541945 seconds. | |
| 2018-05-07 20:23:23,189 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:23,189 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.401518 seconds. | |
| 2018-05-07 20:23:26,594 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:26,595 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.721336 seconds. | |
| 2018-05-07 20:23:26,767 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:26,768 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:26,768 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.357067 seconds. | |
| 2018-05-07 20:23:33,132 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:33,132 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:33,132 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.250864 seconds. | |
| 2018-05-07 20:23:33,318 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:33,318 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.131657 seconds. | |
| 2018-05-07 20:23:37,387 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:37,388 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs. | |
| 2018-05-07 20:23:37,388 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.474147 seconds. | |
| 2018-05-07 20:23:38,455 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:38,455 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.824235 seconds. | |
| 2018-05-07 20:23:40,281 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:40,282 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.290993 seconds. | |
| ^C | |
| ~ ⎈ gke-cognitedata-production | |
| $ k logs -f search-loader-5c8fd5bd57-x9xnk | |
| 2018-05-07 20:23:35,025 INFO 76:cognite_logger.py(1) - Live worker enabled. Initializing queue... | |
| 2018-05-07 20:23:35,033 INFO 76:cognite_logger.py(1) - Backfill worker enabled. Initializing queue... | |
| 2018-05-07 20:23:35,039 INFO 76:cognite_logger.py(1) - ElasticsearchLoader initialized, not started yet. | |
| 2018-05-07 20:23:35,042 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
| 2018-05-07 20:23:35,093 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
| 2018-05-07 20:23:35,093 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:35,094 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
| 2018-05-07 20:23:35,095 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.730845 seconds. | |
| 2018-05-07 20:23:35,096 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:23:35,096 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/resource-changes-livewith subscription projects/cognitedata-production/subscriptions/search-loader-resource-changes-live | |
| 2018-05-07 20:23:35,097 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
| 2018-05-07 20:23:35,098 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
| 2018-05-07 20:23:35,098 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:35,098 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
| 2018-05-07 20:23:35,099 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/search-loader-backfillwith subscription projects/cognitedata-production/subscriptions/search-loader-backfill | |
| 2018-05-07 20:23:35,098 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.322453 seconds. | |
| 2018-05-07 20:23:35,099 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:23:35,205 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
| 2018-05-07 20:23:35,297 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
| 2018-05-07 20:23:35,299 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
| 2018-05-07 20:23:35,299 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
| 2018-05-07 20:23:35,397 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
| 2018-05-07 20:23:35,402 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
| 2018-05-07 20:23:38,424 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:38,424 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.208393 seconds. | |
| 2018-05-07 20:23:42,835 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:42,835 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.763262 seconds. | |
| 2018-05-07 20:23:46,641 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:46,641 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.059000 seconds. | |
| 2018-05-07 20:23:50,606 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:50,606 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.936351 seconds. | |
| 2018-05-07 20:23:50,705 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:50,705 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.493366 seconds. | |
| 2018-05-07 20:23:51,543 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:51,544 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.370397 seconds. | |
| 2018-05-07 20:23:55,963 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:55,964 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.451475 seconds. | |
| 2018-05-07 20:23:57,417 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:57,418 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.904849 seconds. | |
| 2018-05-07 20:23:58,206 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:58,206 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.443527 seconds. | |
| 2018-05-07 20:23:58,651 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:58,651 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.535724 seconds. | |
| 2018-05-07 20:24:02,191 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:02,191 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.092101 seconds. | |
| 2018-05-07 20:24:02,324 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:02,324 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.445057 seconds. | |
| 2018-05-07 20:24:03,771 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:03,771 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.937518 seconds. | |
| 2018-05-07 20:24:07,288 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:07,289 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.989881 seconds. | |
| 2018-05-07 20:24:07,713 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:07,713 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.826623 seconds. | |
| 2018-05-07 20:24:09,542 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:09,542 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.622086 seconds. | |
| 2018-05-07 20:24:10,165 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:10,165 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.280877 seconds. | |
| 2018-05-07 20:24:11,282 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:11,282 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.222935 seconds. | |
| 2018-05-07 20:24:14,448 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:14,449 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.875739 seconds. | |
| 2018-05-07 20:24:16,511 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:16,511 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.144540 seconds. | |
| 2018-05-07 20:24:21,661 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:21,661 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.476803 seconds. | |
| 2018-05-07 20:24:23,325 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:23,326 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.667869 seconds. | |
| 2018-05-07 20:24:23,994 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:23,995 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.218226 seconds. | |
| 2018-05-07 20:24:26,143 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:26,143 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.536331 seconds. | |
| 2018-05-07 20:24:26,215 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:26,215 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.604584 seconds. | |
| 2018-05-07 20:24:26,821 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:26,821 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.705008 seconds. | |
| 2018-05-07 20:24:27,681 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:27,682 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.322410 seconds. | |
| 2018-05-07 20:24:31,531 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:31,532 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.477748 seconds. | |
| 2018-05-07 20:24:34,011 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:34,011 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.743311 seconds. | |
| 2018-05-07 20:24:35,014 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:35,014 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.153356 seconds. | |
| 2018-05-07 20:24:41,173 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:41,174 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.699673 seconds. | |
| 2018-05-07 20:24:41,874 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:41,875 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.206370 seconds. | |
| 2018-05-07 20:24:42,763 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:42,764 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.274700 seconds. | |
| 2018-05-07 20:24:45,041 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:45,041 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.749096 seconds. | |
| 2018-05-07 20:24:46,082 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:46,082 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.384495 seconds. | |
| 2018-05-07 20:24:48,248 DEBUG 440:bidi.py(1) - recved response. | |
| 2018-05-07 20:24:48,248 DEBUG 340:streaming_pull_manager.py(1) - Scheduling callbacks for 1 messages. | |
| 2018-05-07 20:24:48,249 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
| 2018-05-07 20:24:48,293 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:24:48,305 DEBUG 71:cognite_logger.py(1) - b'ghjkl' | |
| 2018-05-07 20:24:48,311 ERROR 86:cognite_logger.py(1) - Could not parse message, getting rid of it. Expecting value: line 1 column 1 (char 0) | |
| Traceback (most recent call last): | |
| File "/cognite_search_loader/loader.py", line 120, in process_live_message | |
| data = json.loads(message.data) | |
| File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads | |
| return _default_decoder.decode(s) | |
| File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode | |
| obj, end = self.raw_decode(s, idx=_w(s, 0).end()) | |
| File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode | |
| raise JSONDecodeError("Expecting value", s, err.value) from None | |
| json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) | |
| 2018-05-07 20:24:48,312 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
| 2018-05-07 20:24:48,793 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:48,794 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.272399 seconds. | |
| 2018-05-07 20:24:52,070 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:52,070 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.218485 seconds. | |
| 2018-05-07 20:24:54,471 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:54,472 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.696924 seconds. | |
| 2018-05-07 20:24:55,292 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:55,292 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.860597 seconds. | |
| 2018-05-07 20:24:55,431 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
| 2018-05-07 20:25:01,175 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:01,175 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.276403 seconds. | |
| 2018-05-07 20:25:02,155 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:02,155 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.027589 seconds. | |
| 2018-05-07 20:25:03,454 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:03,454 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.777448 seconds. | |
| 2018-05-07 20:25:07,185 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:07,185 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.296408 seconds. | |
| 2018-05-07 20:25:08,233 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:08,233 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.980049 seconds. | |
| 2018-05-07 20:25:12,483 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:12,484 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.068263 seconds. | |
| 2018-05-07 20:25:14,554 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:14,554 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.770624 seconds. | |
| 2018-05-07 20:25:17,223 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:17,223 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.750148 seconds. | |
| 2018-05-07 20:25:20,977 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:20,978 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.130011 seconds. | |
| 2018-05-07 20:25:21,327 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:21,327 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.685115 seconds. | |
| 2018-05-07 20:25:22,013 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:22,014 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.288631 seconds. | |
| 2018-05-07 20:25:24,303 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:24,304 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.271615 seconds. | |
| 2018-05-07 20:25:25,112 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:25,112 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.278409 seconds. | |
| 2018-05-07 20:25:27,579 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:27,579 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.451800 seconds. | |
| 2018-05-07 20:25:29,394 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:29,394 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.617179 seconds. | |
| 2018-05-07 20:25:34,016 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:34,017 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.046940 seconds. | |
| 2018-05-07 20:25:34,037 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:34,037 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.474646 seconds. | |
| 2018-05-07 20:25:37,066 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:37,066 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.801040 seconds. | |
| 2018-05-07 20:25:38,514 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:38,514 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.342559 seconds. | |
| 2018-05-07 20:25:38,857 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:38,857 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.864900 seconds. | |
| 2018-05-07 20:25:45,873 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:45,874 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.157340 seconds. | |
| 2018-05-07 20:25:46,724 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:46,724 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.036529 seconds. | |
| 2018-05-07 20:25:47,033 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:47,033 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.626737 seconds. | |
| ^C | |
| ~ ⎈ gke-cognitedata-production | |
| $ k logs -f search-loader-5c8fd5bd57-x9xnk | |
| 2018-05-07 20:23:35,025 INFO 76:cognite_logger.py(1) - Live worker enabled. Initializing queue... | |
| 2018-05-07 20:23:35,033 INFO 76:cognite_logger.py(1) - Backfill worker enabled. Initializing queue... | |
| 2018-05-07 20:23:35,039 INFO 76:cognite_logger.py(1) - ElasticsearchLoader initialized, not started yet. | |
| 2018-05-07 20:23:35,042 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
| 2018-05-07 20:23:35,093 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
| 2018-05-07 20:23:35,093 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:35,094 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
| 2018-05-07 20:23:35,095 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.730845 seconds. | |
| 2018-05-07 20:23:35,096 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:23:35,096 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/resource-changes-livewith subscription projects/cognitedata-production/subscriptions/search-loader-resource-changes-live | |
| 2018-05-07 20:23:35,097 DEBUG 59:dispatcher.py(1) - Started helper thread Thread-CallbackRequestDispatcher | |
| 2018-05-07 20:23:35,098 DEBUG 466:bidi.py(1) - Started helper thread Thread-ConsumeBidirectionalStream | |
| 2018-05-07 20:23:35,098 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:35,098 DEBUG 178:leaser.py(1) - Started helper thread Thread-LeaseMaintainer | |
| 2018-05-07 20:23:35,099 INFO 76:cognite_logger.py(1) - Started loader queue projects/cognitedata-production/topics/search-loader-backfillwith subscription projects/cognitedata-production/subscriptions/search-loader-backfill | |
| 2018-05-07 20:23:35,098 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.322453 seconds. | |
| 2018-05-07 20:23:35,099 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:23:35,205 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
| 2018-05-07 20:23:35,297 DEBUG 117:requests.py(1) - Making request: POST https://accounts.google.com/o/oauth2/token | |
| 2018-05-07 20:23:35,299 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
| 2018-05-07 20:23:35,299 DEBUG 824:connectionpool.py(1) - Starting new HTTPS connection (1): accounts.google.com | |
| 2018-05-07 20:23:35,397 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
| 2018-05-07 20:23:35,402 DEBUG 396:connectionpool.py(1) - https://accounts.google.com:443 "POST /o/oauth2/token HTTP/1.1" 200 None | |
| 2018-05-07 20:23:38,424 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:38,424 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.208393 seconds. | |
| 2018-05-07 20:23:42,835 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:42,835 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.763262 seconds. | |
| 2018-05-07 20:23:46,641 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:46,641 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.059000 seconds. | |
| 2018-05-07 20:23:50,606 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:50,606 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.936351 seconds. | |
| 2018-05-07 20:23:50,705 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:50,705 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.493366 seconds. | |
| 2018-05-07 20:23:51,543 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:51,544 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.370397 seconds. | |
| 2018-05-07 20:23:55,963 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:55,964 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.451475 seconds. | |
| 2018-05-07 20:23:57,417 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:57,418 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.904849 seconds. | |
| 2018-05-07 20:23:58,206 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:58,206 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.443527 seconds. | |
| 2018-05-07 20:23:58,651 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:23:58,651 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.535724 seconds. | |
| 2018-05-07 20:24:02,191 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:02,191 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.092101 seconds. | |
| 2018-05-07 20:24:02,324 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:02,324 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.445057 seconds. | |
| 2018-05-07 20:24:03,771 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:03,771 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.937518 seconds. | |
| 2018-05-07 20:24:07,288 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:07,289 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.989881 seconds. | |
| 2018-05-07 20:24:07,713 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:07,713 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.826623 seconds. | |
| 2018-05-07 20:24:09,542 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:09,542 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.622086 seconds. | |
| 2018-05-07 20:24:10,165 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:10,165 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.280877 seconds. | |
| 2018-05-07 20:24:11,282 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:11,282 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.222935 seconds. | |
| 2018-05-07 20:24:14,448 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:14,449 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.875739 seconds. | |
| 2018-05-07 20:24:16,511 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:16,511 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.144540 seconds. | |
| 2018-05-07 20:24:21,661 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:21,661 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.476803 seconds. | |
| 2018-05-07 20:24:23,325 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:23,326 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.667869 seconds. | |
| 2018-05-07 20:24:23,994 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:23,995 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.218226 seconds. | |
| 2018-05-07 20:24:26,143 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:26,143 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.536331 seconds. | |
| 2018-05-07 20:24:26,215 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:26,215 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.604584 seconds. | |
| 2018-05-07 20:24:26,821 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:26,821 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.705008 seconds. | |
| 2018-05-07 20:24:27,681 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:27,682 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.322410 seconds. | |
| 2018-05-07 20:24:31,531 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:31,532 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.477748 seconds. | |
| 2018-05-07 20:24:34,011 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:34,011 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.743311 seconds. | |
| 2018-05-07 20:24:35,014 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:35,014 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.153356 seconds. | |
| 2018-05-07 20:24:41,173 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:41,174 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.699673 seconds. | |
| 2018-05-07 20:24:41,874 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:41,875 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.206370 seconds. | |
| 2018-05-07 20:24:42,763 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:42,764 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.274700 seconds. | |
| 2018-05-07 20:24:45,041 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:45,041 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.749096 seconds. | |
| 2018-05-07 20:24:46,082 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:46,082 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.384495 seconds. | |
| 2018-05-07 20:24:48,248 DEBUG 440:bidi.py(1) - recved response. | |
| 2018-05-07 20:24:48,248 DEBUG 340:streaming_pull_manager.py(1) - Scheduling callbacks for 1 messages. | |
| 2018-05-07 20:24:48,249 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
| 2018-05-07 20:24:48,293 DEBUG 438:bidi.py(1) - waiting for recv. | |
| 2018-05-07 20:24:48,305 DEBUG 71:cognite_logger.py(1) - b'ghjkl' | |
| 2018-05-07 20:24:48,311 ERROR 86:cognite_logger.py(1) - Could not parse message, getting rid of it. Expecting value: line 1 column 1 (char 0) | |
| Traceback (most recent call last): | |
| File "/cognite_search_loader/loader.py", line 120, in process_live_message | |
| data = json.loads(message.data) | |
| File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads | |
| return _default_decoder.decode(s) | |
| File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode | |
| obj, end = self.raw_decode(s, idx=_w(s, 0).end()) | |
| File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode | |
| raise JSONDecodeError("Expecting value", s, err.value) from None | |
| json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) | |
| 2018-05-07 20:24:48,312 DEBUG 91:dispatcher.py(1) - Handling 1 batched requests | |
| 2018-05-07 20:24:48,793 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:48,794 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.272399 seconds. | |
| 2018-05-07 20:24:52,070 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:52,070 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.218485 seconds. | |
| 2018-05-07 20:24:54,471 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:54,472 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.696924 seconds. | |
| 2018-05-07 20:24:55,292 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:24:55,292 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.860597 seconds. | |
| 2018-05-07 20:24:55,431 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
| 2018-05-07 20:25:01,175 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:01,175 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.276403 seconds. | |
| 2018-05-07 20:25:02,155 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:02,155 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.027589 seconds. | |
| 2018-05-07 20:25:03,454 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:03,454 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.777448 seconds. | |
| 2018-05-07 20:25:07,185 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:07,185 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.296408 seconds. | |
| 2018-05-07 20:25:08,233 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:08,233 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.980049 seconds. | |
| 2018-05-07 20:25:12,483 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:12,484 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.068263 seconds. | |
| 2018-05-07 20:25:14,554 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:14,554 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.770624 seconds. | |
| 2018-05-07 20:25:17,223 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:17,223 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.750148 seconds. | |
| 2018-05-07 20:25:20,977 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:20,978 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.130011 seconds. | |
| 2018-05-07 20:25:21,327 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:21,327 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.685115 seconds. | |
| 2018-05-07 20:25:22,013 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:22,014 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.288631 seconds. | |
| 2018-05-07 20:25:24,303 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:24,304 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.271615 seconds. | |
| 2018-05-07 20:25:25,112 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:25,112 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.278409 seconds. | |
| 2018-05-07 20:25:27,579 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:27,579 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.451800 seconds. | |
| 2018-05-07 20:25:29,394 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:29,394 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.617179 seconds. | |
| 2018-05-07 20:25:34,016 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:34,017 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.046940 seconds. | |
| 2018-05-07 20:25:34,037 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:34,037 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.474646 seconds. | |
| 2018-05-07 20:25:37,066 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:37,066 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.801040 seconds. | |
| 2018-05-07 20:25:38,514 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:38,514 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.342559 seconds. | |
| 2018-05-07 20:25:38,857 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:38,857 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.864900 seconds. | |
| 2018-05-07 20:25:45,873 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:45,874 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.157340 seconds. | |
| 2018-05-07 20:25:46,724 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:46,724 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.036529 seconds. | |
| 2018-05-07 20:25:47,033 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:47,033 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.626737 seconds. | |
| 2018-05-07 20:25:52,766 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:52,766 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.108176 seconds. | |
| 2018-05-07 20:25:52,875 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:52,875 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.432782 seconds. | |
| 2018-05-07 20:25:55,663 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:55,664 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.603902 seconds. | |
| 2018-05-07 20:25:56,310 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:25:56,311 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.093734 seconds. | |
| 2018-05-07 20:25:57,377 DEBUG 448:bidi.py(1) - Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code. | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next | |
| return six.next(self._wrapped) | |
| File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__ | |
| return self._next() | |
| File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next | |
| raise self | |
| grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])> | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 348, in _recoverable | |
| return method(*args, **kwargs) | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv | |
| return next(self.call) | |
| File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next | |
| six.raise_from(exceptions.from_grpc_error(exc), exc) | |
| File "<string>", line 3, in raise_from | |
| google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75] | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next | |
| return six.next(self._wrapped) | |
| File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__ | |
| return self._next() | |
| File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next | |
| raise self | |
| grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])> | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 439, in _thread_main | |
| response = self._bidi_rpc.recv() | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 365, in recv | |
| super(ResumableBidiRpc, self).recv) | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 357, in _recoverable | |
| return method(*args, **kwargs) | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv | |
| return next(self.call) | |
| File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next | |
| six.raise_from(exceptions.from_grpc_error(exc), exc) | |
| File "<string>", line 3, in raise_from | |
| google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75] | |
| 2018-05-07 20:25:57,378 INFO 455:bidi.py(1) - Thread-ConsumeBidirectionalStream exiting | |
| 2018-05-07 20:25:58,108 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
| 2018-05-07 20:25:58,407 INFO 164:leaser.py(1) - Thread-LeaseMaintainer exiting. | |
| 2018-05-07 20:26:02,275 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:02,275 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.549899 seconds. | |
| 2018-05-07 20:26:09,833 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:09,833 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.363373 seconds. | |
| 2018-05-07 20:26:10,197 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:10,198 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.908064 seconds. | |
| 2018-05-07 20:26:14,110 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:14,110 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.179245 seconds. | |
| 2018-05-07 20:26:21,296 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:21,296 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.171606 seconds. | |
| 2018-05-07 20:26:25,398 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. | |
| 2018-05-07 20:26:26,473 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:26,474 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.733286 seconds. | |
| 2018-05-07 20:26:32,213 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:32,214 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.333760 seconds. | |
| 2018-05-07 20:26:38,552 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:38,552 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.954605 seconds. | |
| 2018-05-07 20:26:45,514 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:45,514 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.039095 seconds. | |
| 2018-05-07 20:26:53,555 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:53,555 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.136897 seconds. | |
| 2018-05-07 20:26:53,693 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:26:53,693 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.990966 seconds. | |
| 2018-05-07 20:27:02,689 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:02,689 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.007966 seconds. | |
| 2018-05-07 20:27:06,702 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:06,702 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.666940 seconds. | |
| 2018-05-07 20:27:14,373 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:14,374 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.576805 seconds. | |
| 2018-05-07 20:27:21,958 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:21,959 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.749982 seconds. | |
| 2018-05-07 20:27:27,715 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:27,715 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.272717 seconds. | |
| 2018-05-07 20:27:30,991 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:30,991 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.771143 seconds. | |
| 2018-05-07 20:27:33,763 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:33,763 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.827710 seconds. | |
| 2018-05-07 20:27:34,592 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:34,592 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.229114 seconds. | |
| 2018-05-07 20:27:34,822 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:34,822 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.949239 seconds. | |
| 2018-05-07 20:27:43,773 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:43,774 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.192543 seconds. | |
| 2018-05-07 20:27:50,971 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:50,971 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.647281 seconds. | |
| 2018-05-07 20:27:59,623 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:27:59,624 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.893704 seconds. | |
| 2018-05-07 20:28:03,522 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:28:03,522 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.280092 seconds. | |
| 2018-05-07 20:28:10,810 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:28:10,810 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.781080 seconds. | |
| 2018-05-07 20:28:18,594 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds. | |
| 2018-05-07 20:28:18,595 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.834090 seconds. | |
| 2018-05-07 20:28:19,308 DEBUG 448:bidi.py(1) - Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code. | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next | |
| return six.next(self._wrapped) | |
| File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__ | |
| return self._next() | |
| File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next | |
| raise self | |
| grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])> | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 348, in _recoverable | |
| return method(*args, **kwargs) | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv | |
| return next(self.call) | |
| File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next | |
| six.raise_from(exceptions.from_grpc_error(exc), exc) | |
| File "<string>", line 3, in raise_from | |
| google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75] | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next | |
| return six.next(self._wrapped) | |
| File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__ | |
| return self._next() | |
| File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next | |
| raise self | |
| grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])> | |
| The above exception was the direct cause of the following exception: | |
| Traceback (most recent call last): | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 439, in _thread_main | |
| response = self._bidi_rpc.recv() | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 365, in recv | |
| super(ResumableBidiRpc, self).recv) | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 357, in _recoverable | |
| return method(*args, **kwargs) | |
| File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv | |
| return next(self.call) | |
| File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next | |
| six.raise_from(exceptions.from_grpc_error(exc), exc) | |
| File "<string>", line 3, in raise_from | |
| google.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75] | |
| 2018-05-07 20:28:19,309 INFO 455:bidi.py(1) - Thread-ConsumeBidirectionalStream exiting | |
| 2018-05-07 20:28:19,429 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator. |
Another error
2018-05-07 21:26:56,869 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.056173 seconds.
2018-05-07 21:27:03,933 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:03,933 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:03,933 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.459262 seconds.
2018-05-07 21:27:10,399 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:10,400 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:10,400 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.273510 seconds.
2018-05-07 21:27:17,681 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:17,681 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:17,681 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.661820 seconds.
2018-05-07 21:27:25,351 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:25,351 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:25,352 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.279241 seconds.
2018-05-07 21:27:25,631 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:25,632 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:25,632 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.759359 seconds.
2018-05-07 21:27:32,397 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:32,398 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:32,398 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.300170 seconds.
2018-05-07 21:27:33,700 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:33,700 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:33,700 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.791485 seconds.
2018-05-07 21:27:35,494 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:35,494 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:35,494 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.287875 seconds.
2018-05-07 21:27:37,783 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:37,784 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:37,784 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.582993 seconds.
2018-05-07 21:27:42,372 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:42,372 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:42,372 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.168857 seconds.
2018-05-07 21:27:42,541 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:42,542 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:42,542 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.407904 seconds.
2018-05-07 21:27:42,950 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:42,951 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:42,951 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.896689 seconds.
2018-05-07 21:27:48,854 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:48,854 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:48,854 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.278078 seconds.
2018-05-07 21:27:56,140 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:27:56,140 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:27:56,141 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.239156 seconds.
2018-05-07 21:28:03,387 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:03,387 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:03,388 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.519998 seconds.
2018-05-07 21:28:09,914 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:09,915 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:09,915 DEBUG 161:leaser.py(1) - Snoozing lease management for 8.722808 seconds.
2018-05-07 21:28:18,642 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:18,642 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:18,642 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.504670 seconds.
2018-05-07 21:28:21,150 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:21,150 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:21,150 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.186472 seconds.
2018-05-07 21:28:22,338 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:22,338 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:22,339 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.917794 seconds.
2018-05-07 21:28:23,258 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:23,258 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:23,258 DEBUG 161:leaser.py(1) - Snoozing lease management for 2.410753 seconds.
2018-05-07 21:28:25,672 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:25,672 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:25,672 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.307804 seconds.
2018-05-07 21:28:28,984 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:28,984 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:28,984 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.991359 seconds.
2018-05-07 21:28:32,980 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:32,980 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:32,981 DEBUG 161:leaser.py(1) - Snoozing lease management for 5.906401 seconds.
2018-05-07 21:28:38,892 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:38,892 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:38,892 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.522132 seconds.
2018-05-07 21:28:40,415 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:40,416 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:40,416 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.013081 seconds.
2018-05-07 21:28:44,433 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:44,434 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:44,434 DEBUG 161:leaser.py(1) - Snoozing lease management for 0.886840 seconds.
2018-05-07 21:28:45,379 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:45,380 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:45,380 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.678250 seconds.
2018-05-07 21:28:47,060 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:47,061 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:47,061 DEBUG 161:leaser.py(1) - Snoozing lease management for 7.012556 seconds.
2018-05-07 21:28:54,081 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:54,081 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:54,081 DEBUG 161:leaser.py(1) - Snoozing lease management for 3.841112 seconds.
2018-05-07 21:28:57,926 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:28:57,927 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:28:57,927 DEBUG 161:leaser.py(1) - Snoozing lease management for 4.534431 seconds.
2018-05-07 21:29:02,527 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:29:02,528 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:29:02,528 DEBUG 161:leaser.py(1) - Snoozing lease management for 1.860121 seconds.
2018-05-07 21:29:04,452 DEBUG 108:leaser.py(1) - The current p99 value is 10 seconds.
2018-05-07 21:29:04,452 DEBUG 144:leaser.py(1) - Renewing lease for 1 ack IDs.
2018-05-07 21:29:04,452 DEBUG 161:leaser.py(1) - Snoozing lease management for 6.452385 seconds.
2018-05-07 21:29:09,921 DEBUG 448:bidi.py(1) - Thread-ConsumeBidirectionalStream caught error 504 Deadline Exceeded and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next
return six.next(self._wrapped)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__
return self._next()
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded)>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 348, in _recoverable
return method(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv
return next(self.call)
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 76, in next
return six.next(self._wrapped)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 347, in __next__
return self._next()
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 341, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded)>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 439, in _thread_main
response = self._bidi_rpc.recv()
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 365, in recv
super(ResumableBidiRpc, self).recv)
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 357, in _recoverable
return method(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/bidi.py", line 258, in recv
return next(self.call)
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 78, in next
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded
2018-05-07 21:29:09,922 INFO 455:bidi.py(1) - Thread-ConsumeBidirectionalStream exiting
2018-05-07 21:29:10,492 DEBUG 110:bidi.py(1) - Empty queue and inactive call, exiting request generator.
2018-05-07 21:29:10,949 INFO 164:leaser.py(1) - Thread-LeaseMaintainer exiting.
hi,
What's the fix for the above error?
I am also getting a similar error, when there is no message in PubSub:
code:
while True:
response = subscriber.api.pull(sub_path, max_messages=int(_NUM_MESSAGES))
---------
sleep(0.1)
Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 549, in call
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
that was a long time ago, and I think it was fixed by upgrading to newer versions of the pubsub library. We are not using the Python sdk anymore, can't help.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With a single consumer: