Skip to content

Conversation

@Dawars
Copy link
Contributor

@Dawars Dawars commented May 12, 2019

I started adding support for the new mesh/point cloud data type introduced to TensorBoard recently.

I created the functions to add the data, created the appropriate summaries.
This new data type however requires a Merged summary containing the data for the vertices, colors and faces.

I got stuck at this stage. Maybe someone can help. @lanpa?

I converted the example code by Google to PyTorch:

import numpy as np
import trimesh

import torch
from torch.utils.tensorboard import SummaryWriter

sample_mesh = 'https://storage.googleapis.com/tensorflow-graphics/tensorboard/test_data/ShortDance07_a175_00001.ply'
log_dir = 'runs/torch'
batch_size = 1

# Camera and scene configuration.
config_dict = {
    'camera': {'cls': 'PerspectiveCamera', 'fov': 75},
    'lights': [
        {
            'cls': 'AmbientLight',
            'color': '#ffffff',
            'intensity': 0.75,
        }, {
            'cls': 'DirectionalLight',
            'color': '#ffffff',
            'intensity': 0.75,
            'position': [0, -1, 2],
        }],
    'material': {
        'cls': 'MeshStandardMaterial',
        'roughness': 1,
        'metalness': 0
    }
}

# Read all sample PLY files.
mesh = trimesh.load_remote(sample_mesh)
vertices = np.array(mesh.vertices)
# Currently only supports RGB colors.
colors = np.array(mesh.visual.vertex_colors[:, :3])
faces = np.array(mesh.faces)

# Add batch dimension, so our data will be of shape BxNxC.
vertices = np.expand_dims(vertices, 0)
colors = np.expand_dims(colors, 0)
faces = np.expand_dims(faces, 0)

# Create data placeholders of the same shape as data itself.
vertices_tensor = torch.as_tensor(vertices)
faces_tensor = torch.as_tensor(faces)
colors_tensor = torch.as_tensor(colors)

writer = SummaryWriter(log_dir)

writer.add_mesh('mesh_color_tensor', vertices=vertices_tensor, faces=faces_tensor,
                colors=colors_tensor, config_dict=config_dict)

writer.close()

I tried adding only the vertex summary, hence the others are supposed to be optional.
I got the following error from TensorBoard and it also didn't display the points:

Traceback (most recent call last):
  File "/home/dawars/workspace/pytorch/venv/lib/python3.6/site-packages/werkzeug/serving.py", line 302, in run_wsgi
    execute(self.server.app)
  File "/home/dawars/workspace/pytorch/venv/lib/python3.6/site-packages/werkzeug/serving.py", line 290, in execute
    application_iter = app(environ, start_response)
  File "/home/dawars/workspace/pytorch/venv/lib/python3.6/site-packages/tensorboard/backend/application.py", line 309, in __call__
    return self.data_applications[clean_path](environ, start_response)
  File "/home/dawars/workspace/pytorch/venv/lib/python3.6/site-packages/werkzeug/wrappers/base_request.py", line 235, in application
    resp = f(*args[:-2] + (request,))
  File "/home/dawars/workspace/pytorch/venv/lib/python3.6/site-packages/tensorboard/plugins/mesh/mesh_plugin.py", line 252, in _serve_mesh_metadata
    tensor_events = self._collect_tensor_events(request)
  File "/home/dawars/workspace/pytorch/venv/lib/python3.6/site-packages/tensorboard/plugins/mesh/mesh_plugin.py", line 188, in _collect_tensor_events
    tensors = self._multiplexer.Tensors(run, instance_tag)
  File "/home/dawars/workspace/pytorch/venv/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 400, in Tensors
    return accumulator.Tensors(tag)
  File "/home/dawars/workspace/pytorch/venv/lib/python3.6/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 437, in Tensors
    return self.tensors_by_tag[tag].Items(_TENSOR_RESERVOIR_KEY)
KeyError: 'mesh_color_tensor_COLOR'

Summaries containing data added, needs to `merge` them into one.
@jeffreyksmithjr jeffreyksmithjr requested a review from orionr May 13, 2019 22:39
@jeffreyksmithjr jeffreyksmithjr added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label May 13, 2019
@orionr orionr requested a review from lanpa May 13, 2019 23:23
@lanpa
Copy link
Collaborator

lanpa commented May 14, 2019

Hi, which version of tensorboard are you using?

@Dawars
Copy link
Contributor Author

Dawars commented May 14, 2019

The nightly version from pip:
tb-nightly 1.14.0a20190512
tf-nightly-gpu 1.14.1.dev20190512

Copy link
Contributor

@orionr orionr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is excellent! Really exciting and great to see this contribution. We need to add a bit more documentation and cinch up the code a bit, but this is great. @lanpa you might be better equipped to get this in a ready state, but I added some comments here for all of us.

@orionr
Copy link
Contributor

orionr commented May 20, 2019

@Dawars and @lanpa, added some comments here. This is looking great, though. Kudos!

@pytorchbot pytorchbot added the module: ci Related to continuous integration label May 21, 2019
@pytorchbot pytorchbot added the module: docs Related to our documentation, both in docs/ and docblocks label May 22, 2019
@Dawars
Copy link
Contributor Author

Dawars commented May 22, 2019

Oh, I wasn't supposed to merge master just yet, how can I fix this?

@lanpa
Copy link
Collaborator

lanpa commented May 23, 2019

@Dawars That shouldn't break the following merge.
cc @orionr for a review

@orionr orionr changed the title [WIP] TensorBoard 3D mesh/point cloud support [tensorboard] Support 3D mesh/point cloud May 24, 2019
Copy link
Contributor

@orionr orionr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@orionr
Copy link
Contributor

orionr commented May 24, 2019

@pytorchbot retest this please.

Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@orionr is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@orionr
Copy link
Contributor

orionr commented May 24, 2019

Unrelated test failures.

@facebook-github-bot
Copy link
Contributor

@orionr merged this pull request in b5a5e29.

facebook-github-bot pushed a commit that referenced this pull request May 29, 2019
Summary:
I accidentally added a TF dependency in #20413 by using the from tensorboard.plugins.mesh.summary import _get_json_config import.

I'm removing it at the cost of code duplication.

orionr, Please review.
Pull Request resolved: #21066

Reviewed By: natalialunova

Differential Revision: D15538746

Pulled By: orionr

fbshipit-source-id: 8a822719a4a9f5d67f1badb474e3a73cefce507f
akyrola pushed a commit to akyrola/pytorch that referenced this pull request Jun 3, 2019
Summary:
I accidentally added a TF dependency in pytorch#20413 by using the from tensorboard.plugins.mesh.summary import _get_json_config import.

I'm removing it at the cost of code duplication.

orionr, Please review.
Pull Request resolved: pytorch#21066

Reviewed By: natalialunova

Differential Revision: D15538746

Pulled By: orionr

fbshipit-source-id: 8a822719a4a9f5d67f1badb474e3a73cefce507f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Merged module: ci Related to continuous integration module: docs Related to our documentation, both in docs/ and docblocks module: tensorboard open source triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants