Make putting large objects work. #411

pcmoritz · 2017-03-29T20:49:21Z

This should address #412.

robertnishihara · 2017-03-30T00:12:47Z

Let's hold off on merging this due to https://issues.apache.org/jira/browse/ARROW-739. Although actually this problem may already be present in the Arrow commit in the current master.

robertnishihara · 2017-03-31T19:51:22Z

Since https://issues.apache.org/jira/browse/ARROW-739 was addressed by apache/arrow@ad8a0cf, can you change this patch to use the most recent arrow commit?

AmplabJenkins · 2017-04-01T23:27:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/470/
Test PASSed.

AmplabJenkins · 2017-04-02T20:36:43Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-04-02T20:36:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/477/
Test PASSed.

AmplabJenkins · 2017-04-02T21:12:21Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-04-02T21:12:21Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/478/
Test PASSed.

robertnishihara · 2017-04-04T19:11:25Z

test/jenkins_tests/run_multi_node_tests.sh

    --docker-image=$DOCKER_SHA \
    --num-nodes=5 \
    --test-script=/ray/test/jenkins_tests/multi_node_tests/test_0.py
+


The problem with the tests in Jenkins may just be that we were starting the docker image with a small amount of shared memory (1GB), so maybe it would be sufficient to just increase that number.

AmplabJenkins · 2017-04-05T04:08:26Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-04-05T04:08:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/497/
Test PASSed.

AmplabJenkins · 2017-04-05T05:51:32Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-04-05T05:51:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/498/
Test FAILed.

AmplabJenkins · 2017-04-05T06:14:17Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-04-05T06:14:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/499/
Test FAILed.

AmplabJenkins · 2017-04-05T07:04:17Z

Merged build finished. Test FAILed.

AmplabJenkins · 2017-04-05T07:04:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/501/
Test FAILed.

AmplabJenkins · 2017-04-05T07:31:05Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-04-05T07:31:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/503/
Test PASSed.

pcmoritz · 2017-04-05T16:44:54Z

This validates that arrow's large object support apache/arrow#455 and apache/arrow#404 is working for us end-to-end

Co-authored-by: 黑驰 <[email protected]>

pcmoritz changed the title ~~Make putting large objects work~~ [WIP] Make putting large objects work Mar 29, 2017

putting large objects

2140837

pcmoritz force-pushed the large-objects branch from 97a48ed to 2140837 Compare March 30, 2017 01:38

add more checks

fec2447

pcmoritz force-pushed the large-objects branch from cdf683a to fec2447 Compare March 30, 2017 03:10

pcmoritz added 3 commits March 30, 2017 14:34

support large objects

96bd3bf

fix test

6b489c1

fix linting

b6d3115

pcmoritz added 2 commits March 31, 2017 13:29

upgrade to latest arrow version

e864aeb

check malloc return code

1fd1b08

pcmoritz force-pushed the large-objects branch from b914619 to 1fd1b08 Compare March 31, 2017 22:10

pcmoritz added 2 commits March 31, 2017 15:45

print mmap file sizes

17d223d

printing

5159065

robertnishihara mentioned this pull request Apr 1, 2017

[WIP] Integrate in-memory object store into arrow apache/arrow#467

Closed

pcmoritz added 13 commits April 1, 2017 13:30

revert to dlmalloc

47f4fe6

add prints

0264cac

more prints

5306ab5

add printing

6aba98f

printing

e1c9580

fix

2339edf

update

2842eda

fix

1a4b6f9

update

4637436

print

d7564c1

initialization

dc6c7b0

temp

7b2f4ac

fix

e8ec99c

fix test

1625dc8

evict objects if dlmalloc fails

5be1a44

robertnishihara reviewed Apr 4, 2017

View reviewed changes

fix stresstests

2ee41da

robertnishihara added 3 commits April 4, 2017 21:35

Fix linting.

2d60ff5

Uncomment large-memory tests.

058aa8d

Increase memory for docker image for jenkins tests.

e18f919

robertnishihara changed the title ~~[WIP] Make putting large objects work~~ Make putting large objects work Apr 5, 2017

robertnishihara changed the title ~~Make putting large objects work~~ Make putting large objects work. Apr 5, 2017

Reduce large memory tests.

cd8853a

Further reduce large memory tests.

d82219d

robertnishihara merged commit 4043769 into ray-project:master Apr 5, 2017

robertnishihara deleted the large-objects branch April 5, 2017 08:04

robertnishihara mentioned this pull request Apr 5, 2017

Enable serializing large objects. #412

Closed

Myasuka pushed a commit to Myasuka/ray that referenced this pull request Aug 11, 2025

[2/N][VirtualClusetr] add GcsVirtualClusterTable (ray-project#411)

153b731

Co-authored-by: 黑驰 <[email protected]>

Make putting large objects work. #411

Make putting large objects work. #411

Uh oh!

Conversation

pcmoritz commented Mar 29, 2017 • edited by robertnishihara Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertnishihara commented Mar 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertnishihara commented Mar 31, 2017

Uh oh!

AmplabJenkins commented Apr 1, 2017

Uh oh!

AmplabJenkins commented Apr 2, 2017

Uh oh!

AmplabJenkins commented Apr 2, 2017

Uh oh!

AmplabJenkins commented Apr 2, 2017

Uh oh!

AmplabJenkins commented Apr 2, 2017

Uh oh!

robertnishihara Apr 4, 2017

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

AmplabJenkins commented Apr 5, 2017

Uh oh!

pcmoritz commented Apr 5, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pcmoritz commented Mar 29, 2017 •

edited by robertnishihara

Loading

robertnishihara commented Mar 30, 2017 •

edited

Loading