PyTorch File Format API #9900
PyTorch File Format API #9900
Conversation
jamesr66a
requested review from
apaszke,
colesbury,
ezyang,
gchanan,
soumith and
zdevito
as code owners
jamesr66a
force-pushed
the
file_format_simple
branch
4 times, most recently
from
17dd7bc to
024f7c9
Compare
jamesr66a
force-pushed
the
file_format_simple
branch
from
024f7c9 to
a3713e0
Compare
zdevito
left a comment
zdevito
left a comment
There was a problem hiding this comment.
This looks good. Is there a reason we cannot return std::shared_ptr?
torch/csrc/jit/serialization.h
Outdated
| std::string getLastRecord() { | ||
| return getRecordWithKey(last_record_offset); | ||
| } | ||
| std::string getRecordWithKey(uint64_t key) { |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
torch/csrc/jit/serialization.h
Outdated
| return getRecordWithKey(last_record_offset); | ||
| } | ||
| std::string getRecordWithKey(uint64_t key) { | ||
| if (key > file_size) { |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
torch/csrc/jit/serialization.h
Outdated
|
|
||
| void writePad(const size_t num_bytes) { | ||
| uint8_t pad_val = kPadValue; | ||
| // TODO is there a more efficient way to do this? |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@pytorchbot retest this please |
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
jamesr66a
force-pushed
the
file_format_simple
branch
from
6fbbb04 to
2350ec0
Compare
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@pytorchbot retest this please |
| } | ||
| private: | ||
| FILE *fp; | ||
| size_t cursor = 0; |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
| // Utility functions | ||
| uint64_t read64BitIntegerLittleEndian() { | ||
| uint64_t retval; | ||
| // TODO endian swap on platforms that need it? |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
| std::shared_ptr<void> data; | ||
| size_t size; | ||
| std::tie(data, size) = self.getRecordWithKey(key); | ||
| return py::bytes(reinterpret_cast<const char*>(data.get()), size); |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
| } | ||
| } // namespace | ||
|
|
||
| class PyTorchFileReader { |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
Summary: This is a follow-up to pytorch#9794 that contains only the serialization library and exposes a cleaner API. This should later be incorporated into the module export code Pull Request resolved: pytorch#9900 Reviewed By: zdevito Differential Revision: D9021057 Pulled By: jamesr66a fbshipit-source-id: 01af74a7fdd1b90b2f5484644c3121d8ba9eb3b3
Summary: This is a follow-up to pytorch#9794 that contains only the serialization library and exposes a cleaner API. This should later be incorporated into the module export code Pull Request resolved: pytorch#9900 Reviewed By: zdevito Differential Revision: D9021057 Pulled By: jamesr66a fbshipit-source-id: 01af74a7fdd1b90b2f5484644c3121d8ba9eb3b3
5 participants
This is a follow-up to #9794 that contains only the serialization library and exposes a cleaner API. This should later be incorporated into the module export code