Skip to content

Instantly share code, notes, and snippets.

@nesou2
Created July 17, 2019 10:14
Show Gist options
  • Save nesou2/4e7fa950f48c127d2ba17c34f099b24e to your computer and use it in GitHub Desktop.
Save nesou2/4e7fa950f48c127d2ba17c34f099b24e to your computer and use it in GitHub Desktop.
2019-07-17 11:12:22,333 INFO node.py:498 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-07-17_11-12-22_332372_24448/logs.
2019-07-17 11:12:22,464 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:44759 to respond...
2019-07-17 11:12:22,581 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:40962 to respond...
2019-07-17 11:12:22,583 INFO services.py:806 -- Starting Redis shard with 3.34 GB max memory.
2019-07-17 11:12:22,615 INFO node.py:512 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-07-17_11-12-22_332372_24448/logs.
2019-07-17 11:12:22,617 INFO services.py:1446 -- Starting the Plasma object store with 5.01 GB memory using /dev/shm.
2019-07-17 11:12:22,754 INFO function_runner.py:255 -- tune.track signature detected.
2019-07-17 11:12:22,766 INFO tune.py:61 -- Tip: to resume incomplete experiments, pass resume='prompt' or resume=True to run()
2019-07-17 11:12:22,768 INFO tune.py:233 -- Starting a new experiment.
W0717 11:12:22.897871 140403365500672 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/logger.py:141: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/0 GPUs
Memory usage on this node: 11.0/16.7 GB
2019-07-17 11:12:23,538 WARNING util.py:64 -- The `start_trial` operation took 0.6466984748840332 seconds to complete, which may be a performance bottleneck.
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 1/8 CPUs, 0/0 GPUs
Memory usage on this node: 11.2/16.7 GB
Result logdir: /home/sebastian/ray_results/tune_iris
Number of trials: 1 ({'RUNNING': 1})
RUNNING trials:
- tune_iris_0: RUNNING
(pid=24509) Using TensorFlow backend.
(pid=24509) OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.
(pid=24509) OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
(pid=24509) OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-7
(pid=24509) OMP: Info #156: KMP_AFFINITY: 8 available OS procs
(pid=24509) OMP: Info #157: KMP_AFFINITY: Uniform topology
(pid=24509) OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 2 threads/core (4 total cores)
(pid=24509) OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 0 thread 1
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 1 thread 1
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 2 thread 1
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0
(pid=24509) OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 3 thread 1
(pid=24509) OMP: Info #250: KMP_AFFINITY: pid 24509 tid 24509 thread 0 bound to OS proc set 0
(pid=24509) /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/sklearn/preprocessing/_encoders.py:415: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.
(pid=24509) If you want the future behaviour and silence this warning, you can specify "categories='auto'".
(pid=24509) In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.
(pid=24509) warnings.warn(msg, FutureWarning)
(pid=24509) WARNING: Logging before flag parsing goes to stderr.
(pid=24509) W0717 11:12:25.617969 139634901600000 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
(pid=24509)
(pid=24509) W0717 11:12:25.631628 139634901600000 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
(pid=24509)
(pid=24509) W0717 11:12:25.633145 139634901600000 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
(pid=24509)
(pid=24509) 2019-07-17 11:12:25,647 ERROR function_runner.py:98 -- Runner Thread raised error.
(pid=24509) Traceback (most recent call last):
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 92, in run
(pid=24509) self._entrypoint()
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 143, in entrypoint
(pid=24509) return self._trainable_func(config, self._status_reporter)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 272, in _trainable_func
(pid=24509) output = train_func(config)
(pid=24509) File "<ipython-input-5-e5f3b1eadf8b>", line 6, in tune_iris
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/sequential.py", line 181, in add
(pid=24509) output_tensor = layer(self.outputs[0])
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/base_layer.py", line 431, in __call__
(pid=24509) self.build(unpack_singleton(input_shapes))
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/layers/core.py", line 866, in build
(pid=24509) constraint=self.kernel_constraint)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
(pid=24509) return func(*args, **kwargs)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/base_layer.py", line 249, in add_weight
(pid=24509) weight = K.variable(initializer(shape),
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/initializers.py", line 218, in __call__
(pid=24509) dtype=dtype, seed=self.seed)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 4139, in random_uniform
(pid=24509) dtype=dtype, seed=seed)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 247, in random_uniform
(pid=24509) rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 820, in random_uniform
(pid=24509) name=name)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 626, in _apply_op_helper
(pid=24509) param_name=input_name)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 60, in _SatisfiesTypeConstraint
(pid=24509) ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
(pid=24509) TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64
(pid=24509) Exception in thread Thread-1:
(pid=24509) Traceback (most recent call last):
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 92, in run
(pid=24509) self._entrypoint()
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 143, in entrypoint
(pid=24509) return self._trainable_func(config, self._status_reporter)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 272, in _trainable_func
(pid=24509) output = train_func(config)
(pid=24509) File "<ipython-input-5-e5f3b1eadf8b>", line 6, in tune_iris
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/sequential.py", line 181, in add
(pid=24509) output_tensor = layer(self.outputs[0])
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/base_layer.py", line 431, in __call__
(pid=24509) self.build(unpack_singleton(input_shapes))
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/layers/core.py", line 866, in build
(pid=24509) constraint=self.kernel_constraint)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
(pid=24509) return func(*args, **kwargs)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/engine/base_layer.py", line 249, in add_weight
(pid=24509) weight = K.variable(initializer(shape),
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/initializers.py", line 218, in __call__
(pid=24509) dtype=dtype, seed=self.seed)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 4139, in random_uniform
(pid=24509) dtype=dtype, seed=seed)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 247, in random_uniform
(pid=24509) rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 820, in random_uniform
(pid=24509) name=name)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 626, in _apply_op_helper
(pid=24509) param_name=input_name)
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 60, in _SatisfiesTypeConstraint
(pid=24509) ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
(pid=24509) TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64
(pid=24509)
(pid=24509) During handling of the above exception, another exception occurred:
(pid=24509)
(pid=24509) Traceback (most recent call last):
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/threading.py", line 917, in _bootstrap_inner
(pid=24509) self.run()
(pid=24509) File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 104, in run
(pid=24509) err_tb = err_tb.format_exc()
(pid=24509) AttributeError: 'traceback' object has no attribute 'format_exc'
(pid=24509)
2019-07-17 11:12:26,808 ERROR trial_runner.py:487 -- Error processing event.
Traceback (most recent call last):
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 436, in _process_trial
result = self.trial_executor.fetch_result(trial)
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 323, in fetch_result
result = ray.get(trial_future[0])
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 2195, in get
raise value
ray.exceptions.RayTaskError: ray_WrappedTrackFunc:train() (pid=24509, host=sebastian-debian)
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/trainable.py", line 150, in train
result = self._train()
File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/function_runner.py", line 205, in _train
("Wrapped function ran until completion without reporting "
ray.tune.error.TuneError: Wrapped function ran until completion without reporting results or raising an exception.
2019-07-17 11:12:26,810 INFO ray_trial_executor.py:187 -- Destroying actor for trial tune_iris_0. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/0 GPUs
Memory usage on this node: 11.4/16.7 GB
Result logdir: /home/sebastian/ray_results/tune_iris
Number of trials: 1 ({'ERROR': 1})
ERROR trials:
- tune_iris_0: ERROR, 1 failures: /home/sebastian/ray_results/tune_iris/tune_iris_0_2019-07-17_11-12-220hjrre9i/error_2019-07-17_11-12-26.txt
---------------------------------------------------------------------------
TuneError Traceback (most recent call last)
<ipython-input-5-e5f3b1eadf8b> in <module>
25 config={"lr": 0.1, "dense_1": 1, "dense_2": 0.1},
26 num_samples=1,
---> 27 return_trials=False)
28
29 assert len(results.trials) == 10
~/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/tune.py in run(run_or_experiment, name, stop, config, resources_per_trial, num_samples, local_dir, upload_dir, trial_name_creator, loggers, sync_function, checkpoint_freq, checkpoint_at_end, export_formats, max_failures, restore, search_alg, scheduler, with_server, server_port, verbose, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, return_trials, ray_auto_init)
271 if errored_trials:
272 if raise_on_failed_trial:
--> 273 raise TuneError("Trials did not complete", errored_trials)
274 else:
275 logger.error("Trials did not complete: %s", errored_trials)
TuneError: ('Trials did not complete', [tune_iris_0])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment