-
-
Save nesou2/4e7fa950f48c127d2ba17c34f099b24e to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2019-07-17 11:12:22,333 INFO node.py:498 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-07-17_11-12-22_332372_24448/logs. | |
2019-07-17 11:12:22,464 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:44759 to respond... | |
2019-07-17 11:12:22,581 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:40962 to respond... | |
2019-07-17 11:12:22,583 INFO services.py:806 -- Starting Redis shard with 3.34 GB max memory. | |
2019-07-17 11:12:22,615 INFO node.py:512 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-07-17_11-12-22_332372_24448/logs. | |
2019-07-17 11:12:22,617 INFO services.py:1446 -- Starting the Plasma object store with 5.01 GB memory using /dev/shm. | |
2019-07-17 11:12:22,754 INFO function_runner.py:255 -- tune.track signature detected. | |
2019-07-17 11:12:22,766 INFO tune.py:61 -- Tip: to resume incomplete experiments, pass resume='prompt' or resume=True to run() | |
2019-07-17 11:12:22,768 INFO tune.py:233 -- Starting a new experiment. | |
W0717 11:12:22.897871 140403365500672 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/logger.py:141: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead. | |
== Status == | |
Using FIFO scheduling algorithm. | |
Resources requested: 0/8 CPUs, 0/0 GPUs | |
Memory usage on this node: 11.0/16.7 GB | |
2019-07-17 11:12:23,538 WARNING util.py:64 -- The `start_trial` operation took 0.6466984748840332 seconds to complete, which may be a performance bottleneck. | |
== Status == | |
Using FIFO scheduling algorithm. | |
Resources requested: 1/8 CPUs, 0/0 GPUs | |
Memory usage on this node: 11.2/16.7 GB | |
Result logdir: /home/sebastian/ray_results/tune_iris | |
Number of trials: 1 ({'RUNNING': 1}) | |
RUNNING trials: | |
- tune_iris_0: RUNNING | |
(pid=24509) Using TensorFlow backend. | |
(pid=24509) OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids. | |
(pid=24509) OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info | |
(pid=24509) OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-7 | |
(pid=24509) OMP: Info #156: KMP_AFFINITY: 8 available OS procs | |
(pid=24509) OMP: Info #157: KMP_AFFINITY: Uniform topology | |
(pid=24509) OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 2 threads/core (4 total cores) | |
(pid=24509) OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map: | |
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 | |
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 0 thread 1 | |
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 | |
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 1 thread 1 | |
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0 | |
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 2 thread 1 | |
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0 | |
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1 | |
(pid=24509) OMP: Info #250: KMP_AFFINITY: pid 24509 tid 24509 thread 0 bound to OS proc set 0 | |
(pid=24509) /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py:415: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values. | |
(pid=24509) If you want the future behaviour and silence this warning, you can specify "categories='auto'". | |
(pid=24509) In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly. | |
(pid=24509) warnings.warn(msg, FutureWarning) | |
(pid=24509) WARNING: Logging before flag parsing goes to stderr. | |
(pid=24509) W0717 11:12:25.617969 139634901600000 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. | |
(pid=24509) | |
(pid=24509) W0717 11:12:25.631628 139634901600000 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. | |
(pid=24509) | |
(pid=24509) W0717 11:12:25.633145 139634901600000 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. | |
(pid=24509) | |
(pid=24509) 2019-07-17 11:12:25,647 ERROR function_runner.py:98 -- Runner Thread raised error. | |
(pid=24509) Traceback (most recent call last): | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 92, in run | |
(pid=24509) self._entrypoint() | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 143, in entrypoint | |
(pid=24509) return self._trainable_func(config, self._status_reporter) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 272, in _trainable_func | |
(pid=24509) output = train_func(config) | |
(pid=24509) File "<ipython-input-5-e5f3b1eadf8b>", line 6, in tune_iris | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/sequential.py", line 181, in add | |
(pid=24509) output_tensor = layer(self.outputs[0]) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/base_layer.py", line 431, in __call__ | |
(pid=24509) self.build(unpack_singleton(input_shapes)) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/layers/core.py", line 866, in build | |
(pid=24509) constraint=self.kernel_constraint) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper | |
(pid=24509) return func(*args, **kwargs) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/base_layer.py", line 249, in add_weight | |
(pid=24509) weight = K.variable(initializer(shape), | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/initializers.py", line 218, in __call__ | |
(pid=24509) dtype=dtype, seed=self.seed) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 4139, in random_uniform | |
(pid=24509) dtype=dtype, seed=seed) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 247, in random_uniform | |
(pid=24509) rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 820, in random_uniform | |
(pid=24509) name=name) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 626, in _apply_op_helper | |
(pid=24509) param_name=input_name) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 60, in _SatisfiesTypeConstraint | |
(pid=24509) ", ".join(dtypes.as_dtype(x).name for x in allowed_list))) | |
(pid=24509) TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64 | |
(pid=24509) Exception in thread Thread-1: | |
(pid=24509) Traceback (most recent call last): | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 92, in run | |
(pid=24509) self._entrypoint() | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 143, in entrypoint | |
(pid=24509) return self._trainable_func(config, self._status_reporter) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 272, in _trainable_func | |
(pid=24509) output = train_func(config) | |
(pid=24509) File "<ipython-input-5-e5f3b1eadf8b>", line 6, in tune_iris | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/sequential.py", line 181, in add | |
(pid=24509) output_tensor = layer(self.outputs[0]) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/base_layer.py", line 431, in __call__ | |
(pid=24509) self.build(unpack_singleton(input_shapes)) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/layers/core.py", line 866, in build | |
(pid=24509) constraint=self.kernel_constraint) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper | |
(pid=24509) return func(*args, **kwargs) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/base_layer.py", line 249, in add_weight | |
(pid=24509) weight = K.variable(initializer(shape), | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/initializers.py", line 218, in __call__ | |
(pid=24509) dtype=dtype, seed=self.seed) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 4139, in random_uniform | |
(pid=24509) dtype=dtype, seed=seed) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 247, in random_uniform | |
(pid=24509) rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 820, in random_uniform | |
(pid=24509) name=name) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 626, in _apply_op_helper | |
(pid=24509) param_name=input_name) | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 60, in _SatisfiesTypeConstraint | |
(pid=24509) ", ".join(dtypes.as_dtype(x).name for x in allowed_list))) | |
(pid=24509) TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64 | |
(pid=24509) | |
(pid=24509) During handling of the above exception, another exception occurred: | |
(pid=24509) | |
(pid=24509) Traceback (most recent call last): | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/threading.py", line 917, in _bootstrap_inner | |
(pid=24509) self.run() | |
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 104, in run | |
(pid=24509) err_tb = err_tb.format_exc() | |
(pid=24509) AttributeError: 'traceback' object has no attribute 'format_exc' | |
(pid=24509) | |
2019-07-17 11:12:26,808 ERROR trial_runner.py:487 -- Error processing event. | |
Traceback (most recent call last): | |
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 436, in _process_trial | |
result = self.trial_executor.fetch_result(trial) | |
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 323, in fetch_result | |
result = ray.get(trial_future[0]) | |
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 2195, in get | |
raise value | |
ray.exceptions.RayTaskError: ray_WrappedTrackFunc:train() (pid=24509, host=sebastian-debian) | |
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/trainable.py", line 150, in train | |
result = self._train() | |
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 205, in _train | |
("Wrapped function ran until completion without reporting " | |
ray.tune.error.TuneError: Wrapped function ran until completion without reporting results or raising an exception. | |
2019-07-17 11:12:26,810 INFO ray_trial_executor.py:187 -- Destroying actor for trial tune_iris_0. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads. | |
== Status == | |
Using FIFO scheduling algorithm. | |
Resources requested: 0/8 CPUs, 0/0 GPUs | |
Memory usage on this node: 11.4/16.7 GB | |
Result logdir: /home/sebastian/ray_results/tune_iris | |
Number of trials: 1 ({'ERROR': 1}) | |
ERROR trials: | |
- tune_iris_0: ERROR, 1 failures: /home/sebastian/ray_results/tune_iris/tune_iris_0_2019-07-17_11-12-220hjrre9i/error_2019-07-17_11-12-26.txt | |
--------------------------------------------------------------------------- | |
TuneError Traceback (most recent call last) | |
<ipython-input-5-e5f3b1eadf8b> in <module> | |
25 config={"lr": 0.1, "dense_1": 1, "dense_2": 0.1}, | |
26 num_samples=1, | |
---> 27 return_trials=False) | |
28 | |
29 assert len(results.trials) == 10 | |
~/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/tune.py in run(run_or_experiment, name, stop, config, resources_per_trial, num_samples, local_dir, upload_dir, trial_name_creator, loggers, sync_function, checkpoint_freq, checkpoint_at_end, export_formats, max_failures, restore, search_alg, scheduler, with_server, server_port, verbose, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, return_trials, ray_auto_init) | |
271 if errored_trials: | |
272 if raise_on_failed_trial: | |
--> 273 raise TuneError("Trials did not complete", errored_trials) | |
274 else: | |
275 logger.error("Trials did not complete: %s", errored_trials) | |
TuneError: ('Trials did not complete', [tune_iris_0]) | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment