Error in tune tutorial #1

gistfile1.txt

2019-07-16 12:01:05,311	INFO node.py:498 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-07-16_12-01-05_310279_13121/logs.
2019-07-16 12:01:05,432	INFO services.py:409 -- Waiting for redis server at 127.0.0.1:15015 to respond...
2019-07-16 12:01:05,564	INFO services.py:409 -- Waiting for redis server at 127.0.0.1:33035 to respond...
2019-07-16 12:01:05,571	INFO services.py:806 -- Starting Redis shard with 3.34 GB max memory.
2019-07-16 12:01:05,611	INFO node.py:512 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-07-16_12-01-05_310279_13121/logs.
2019-07-16 12:01:05,613	INFO services.py:1446 -- Starting the Plasma object store with 5.01 GB memory using /dev/shm.
2019-07-16 12:01:05,784	INFO function_runner.py:255 -- tune.track signature detected.
2019-07-16 12:01:05,796	INFO tune.py:61 -- Tip: to resume incomplete experiments, pass resume='prompt' or resume=True to run()
2019-07-16 12:01:05,803	INFO tune.py:233 -- Starting a new experiment.
W0716 12:01:05.887903 140230702511872 deprecation_wrapper.py:119] From /home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/logger.py:141: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/0 GPUs
Memory usage on this node: 9.3/16.7 GB

2019-07-16 12:01:06,750	WARNING util.py:64 -- The `start_trial` operation took 0.8681294918060303 seconds to complete, which may be a performance bottleneck.
2019-07-16 12:01:06,793	ERROR worker.py:1672 -- Failed to unpickle actor class 'WrappedTrackFunc' for actor ID c1852d30b3e30be7815a1c9f416cffd608ca9623. Traceback:
Traceback (most recent call last):
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/function_manager.py", line 728, in _load_actor_class_from_gcs
    actor_class = pickle.loads(pickled_class)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/__init__.py", line 6, in <module>
    from ray.tune.tune import run_experiments, run
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/tune.py", line 11, in <module>
    from ray.tune.analysis import ExperimentAnalysis
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 5, in <module>
    from ray.tune.analysis.experiment_analysis import ExperimentAnalysis
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 10, in <module>
    import pandas as pd
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

2019-07-16 12:01:06,796	ERROR trial_runner.py:487 -- Error processing event.
Traceback (most recent call last):
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 436, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 323, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 2195, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2019-07-16 12:01:06,799	INFO ray_trial_executor.py:187 -- Destroying actor for trial tune_iris_0. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
2019-07-16 12:01:06,808	ERROR worker.py:1672 -- Traceback (most recent call last):
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 884, in _process_task
    self.reraise_actor_init_error()
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 229, in reraise_actor_init_error
    raise self.actor_init_error
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 888, in _process_task
    function_name, args)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 800, in _get_arguments_for_execution
    values = self.get_object(object_ids)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 513, in get_object
    final_results = self.retrieve_and_deserialize(plain_object_ids, 0)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 411, in retrieve_and_deserialize
    self.current_job_id)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 247, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 1151, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 1052, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 1003, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 892, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 943, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 841, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 384, in put_object
    self.store_and_register(object_id, value)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 318, in store_and_register
    self.current_job_id))
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 247, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/worker.py", line 1151, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".
  

2019-07-16 12:01:06,809	ERROR worker.py:1672 -- A worker died or was killed while executing task 5cd66b7c3a4572500ec330f8494f6fe1.

== Status ==
Using FIFO scheduling algorithm.
Resources requested: 1/8 CPUs, 0/0 GPUs
Memory usage on this node: 9.5/16.7 GB
Result logdir: /home/sebastian/ray_results/tune_iris
Number of trials: 1 ({'RUNNING': 1})
RUNNING trials:
 - tune_iris_0:	RUNNING

== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/0 GPUs
Memory usage on this node: 9.6/16.7 GB
Result logdir: /home/sebastian/ray_results/tune_iris
Number of trials: 1 ({'ERROR': 1})
ERROR trials:
 - tune_iris_0:	ERROR, 1 failures: /home/sebastian/ray_results/tune_iris/tune_iris_0_2019-07-16_12-01-05l3p233iq/error_2019-07-16_12-01-06.txt

(pid=13184) 2019-07-16 12:01:06,784	ERROR function_manager.py:731 -- Failed to load actor class %s.
(pid=13184) Traceback (most recent call last):
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/function_manager.py", line 728, in _load_actor_class_from_gcs
(pid=13184)     actor_class = pickle.loads(pickled_class)
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/__init__.py", line 6, in <module>
(pid=13184)     from ray.tune.tune import run_experiments, run
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/tune.py", line 11, in <module>
(pid=13184)     from ray.tune.analysis import ExperimentAnalysis
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/analysis/__init__.py", line 5, in <module>
(pid=13184)     from ray.tune.analysis.experiment_analysis import ExperimentAnalysis
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/analysis/experiment_analysis.py", line 10, in <module>
(pid=13184)     import pandas as pd
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
(pid=13184)     from pandas.compat.numpy import *
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
(pid=13184)     import typing
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/typing.py", line 1357, in <module>
(pid=13184)     class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
(pid=13184)   File "/home/sebastian/anaconda3/envs/mlearning/lib/python3.7/site-packages/typing.py", line 1005, in __new__
(pid=13184)     self._abc_registry = extra._abc_registry
(pid=13184) AttributeError: type object 'Callable' has no attribute '_abc_registry'

---------------------------------------------------------------------------
TuneError                                 Traceback (most recent call last)
<ipython-input-5-cff18de40da4> in <module>
     28     tune_iris,
     29     config={"lr": 0.1, "dense_1": 1, "dense_2": 0.1},
---> 30     num_samples=1,
     31     )
     32 

~/anaconda3/envs/mlearning/lib/python3.7/site-packages/ray/tune/tune.py in run(run_or_experiment, name, stop, config, resources_per_trial, num_samples, local_dir, upload_dir, trial_name_creator, loggers, sync_function, checkpoint_freq, checkpoint_at_end, export_formats, max_failures, restore, search_alg, scheduler, with_server, server_port, verbose, resume, queue_trials, reuse_actors, trial_executor, raise_on_failed_trial, return_trials, ray_auto_init)
    271     if errored_trials:
    272         if raise_on_failed_trial:
--> 273             raise TuneError("Trials did not complete", errored_trials)
    274         else:
    275             logger.error("Trials did not complete: %s", errored_trials)

TuneError: ('Trials did not complete', [tune_iris_0])