Currently if the the argument of __rmatmul__ is not a Tensor, PyTorch throws *** TypeError: matmul(): argument 'other' (position 1) must be Tensor, not Vec

Python data model asks that it return NotImplemented in such case. This allows interpreter to dispatch op to the proper implementation.
(ie see https://docs.python.org/3/library/numbers.html#implementing-the-arithmetic-operations)

Current behavior makes it hard to implement custom types. IE

class FactoredMatrix:
  def __init__(self, mat):
    self.mat = mat
  def __matmul__(self, other):
    return 0
  def __rmatmul__(self, other):
    return 1
x = torch.ones((2,2))
print(FactoredMatrix(x) @ x)  # works
print(x @ FactoredMatrix(x))  # fails

cc @ezyang @gchanan @zou3519