Skip to content

Exceptions and ResourceWarnings on ray.init (Jupyter+offline) #10279

New issue
New issue
Closed
Closed
Exceptions and ResourceWarnings on ray.init (Jupyter+offline)#10279
Labels
P3Issue moderate in impact or severitybugSomething that is supposed to be working; but isn't

Description

@austinmw
austinmw
opened on Aug 24, 2020

What is the problem?

I'm running ray in a JupyterLab notebook (AWS SageMaker), and the notebook has no internet connection (for data safety).
(instance type is ml.p3.16xlarge, so 64 vCPU and 488GiB ram).

When I simply do:

import ray
ray.init()

I get the following output:

2020-08-24 01:29:24,153	INFO resource_spec.py:212 -- Starting Ray with 323.83 GiB memory available for workers and up to 142.79 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.err' mode='a' encoding='UTF-8'>
2020-08-24 01:29:24,500	INFO services.py:1148 -- View the Ray dashboard at localhost:8265
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.err' mode='a' encoding='UTF-8'>
{'node_ip_address': '10.54.198.48',
 'redis_address': '10.54.198.48:45400',
 'object_store_address': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504/sockets/plasma_store',
 'raylet_socket_name': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504/sockets/raylet',
 'webui_url': 'localhost:8265',
 'session_dir': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504'}

Do I need to modify any init settings due to the fact that I'm running in Jupyter and/or because I'm offline?

Ray version and other system information (Python version, TensorFlow version, OS):

  • Python 3.6.10
  • ray 0.8.7

Activity

  1. austinmw
    added
    bugSomething that is supposed to be working; but isn't
    triageNeeds triage (eg: priority, bug/not-bug, and owning component)
    on Aug 24, 2020

  2. rkooo567 commented on Aug 28, 2020

    @rkooo567
    rkooo567
    on Aug 28, 2020
    Contributor

    Is there any way what exceptions are ignored in your logs?

  3. rkooo567
    added
    P3Issue moderate in impact or severity
    and removed
    triageNeeds triage (eg: priority, bug/not-bug, and owning component)
    on Aug 28, 2020

  4. edoakes commented on Aug 28, 2020

    @edoakes
    edoakes
    on Aug 28, 2020
    Collaborator

    @austinmw this is a warning that's printed when a Python file object is created and the file isn't closed before the object is destructed. In most versions of Python this won't be printed, but it will if you're using a debug build or pass -W (and maybe some other circumstances, haven't seen it though).

    This is intentional behavior because we need these files to be long-lived to write logs to, so there's nothing to worry about. Unfortunately I can't think of a good way to silence this at the moment. Leaving this issue open for others that might come across it.

  5. cszhu
    added
    pending-cleanup
    on Jun 16, 2025

  6. cszhu commented on Jun 16, 2025

    @cszhu
    cszhu
    on Jun 16, 2025
    Contributor

    Hello! This P3 issue has seen no activity in at least a year. It will be closed in 2 weeks as part of ongoing cleanup efforts.

    Please remove the pending-cleanup label if you believe this issue should remain open.

    Thanks for contributing to Ray!

  7. cszhu
    closed this as completedon Jun 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Issue moderate in impact or severitybugSomething that is supposed to be working; but isn't

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions