Throw on data dependent outputs #1163
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eellison
changed the title
eellison
requested review from
yanboliang,
mlazos,
voznesenskym and
anijain2305
eellison
commented
Sep 8, 2022
There was a problem hiding this comment.
This is being tested here https://github.com/pytorch/torchdynamo/blob/main/test/test_repros.py#L887
voznesenskym
reviewed
Sep 8, 2022
voznesenskym
reviewed
Sep 8, 2022
voznesenskym
reviewed
Sep 8, 2022
voznesenskym
approved these changes
Sep 9, 2022
| torch.optim.ASGD, exp_frame_cnt=(0 if sys.version_info < (3, 8) else 6) | ||
| ) | ||
|
|
||
| # Fails without fake tensor: |
There was a problem hiding this comment.
Interesting. Can you file an issue for this? Cc @mlazos
yanboliang
reviewed
Sep 9, 2022
| @@ -133,6 +133,7 @@ def fn(x): | |||
| res2 = opt_fn(x) | |||
| self.assertTrue(same(res1, res2)) | |||
|
|
|||
| @patch.object(torchdynamo.config, "fake_tensor_propagation", False) | |||
There was a problem hiding this comment.
I'm curious why this failed? This is one of the test case copied from Pytorch CI, where fake tensor is enabled IIRC.
There was a problem hiding this comment.
>>> x = torch.tensor(0)
>>> torch.randint(x, x, x)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: randint() received an invalid combination of arguments - got (Tensor, Tensor, Tensor), but expected one of:
* (int high, tuple of ints size, *, torch.Generator generator, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
* (int high, tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
* (int low, int high, tuple of ints size, *, torch.Generator generator, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
* (int low, int high, tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
the int->torch.tensor(0) conversion causes an error
…ta_depedent_outputs_changes
…ta_depedent_outputs_changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We need to flip on pytorch/pytorch#83567 for correctness. This patch allows us to continue to trace through
.item()by catching the exception.It also switches the optimizers to use
fake_tensors=Falsebecause as explained in pytorch/pytorch#84597 and pytorch/pytorch#93660, the current wrapping of.item()for those cases only defers the_local_scaler_densecall until torch dispatch layer, where fakes tensors have to throw. Those issues lay out a path forward for not graph breaking, but we should get this patch in even with the graph breaks because it's needed for correctness.This is ready for review but it's going to take another day before the changes in pytorch are available in the nightly and the tests will pass.