A few questions

[NiklasGustafsson]

Moved here from 'Issues.' Originally posted by @BernhardGlueck:

Hi !

I hope this is the right venue to ask this but a few questions came up for me, that would be awesome to get an answer to.

Thanks for the great work and finally a project allowing proper deep learning in C#.

Now for my questions:

1. Is the hinted at productization in the ML.NET roadmap of this project happening ?
2. What are the plans for torchscript export as well as Onnx export as they are the big enabled for production usage...
3. What are the plans for distributed training ( the python tools for that would probably not work, so are there any plans to add a new C# implementation of that kind of functionality ? )
4. Are there plans to bring up the other Accelerators like TPUs etc ?

Thanks for your work and time !

[NiklasGustafsson]

1. The integration into ML.NET is ongoing, but not yet ready to be released or even announced.

2. Currently, there are no plans for either TorchScript or ONNX export. Perfect projects for someone to take on, but currently, we will focus on eager-mode scenarios, which we believe are a) the number one priority, and b) good enough for many scenarios, given that the host runtime is .NET.

3. Another thing that will have to wait a bit, since we need to prioritize the work to keep up with Pytorch as well as the integration with ML.NET. We will probably not tackle distributed training without also tackling network architecture search at the same time.

4. No current plans that we are ready to announce. There are a few things that are necessary -- support for device strings in the main library, and a libtorch_tpu NuGet package. TorchSharp itself is only affected by the former, so there is a little bit of work, but not a lot. If anyone is interested in putting together (and maintaining) a TPU backend, that would be very interesting, and we could relatively easily to the TorchSharp work to support it, but we won't prioritize that here.

If there are other accelerators that you have in mind, like Apple M1, perhaps, please let us know what you would like to see.

[BernhardGlueck]

Thanks for your answers.

We are very eager to finally ditch Python from our codebase for our models, since at a certain code size it just becomes quite cumbersome and the whole ecosystem is a bit .... fiddly to say the least.

Just to give some feedback from our side:

1. The ML.NET integration is actually not really our concern, we are mainly interested in using Torchsharp directly, porting our training code over

2. We basically train large scale vision models that get deployed on edge devices, hence my question... Generally speaking i see very limited use cases for realtime vision models ( video etc.. ) that does NOT run on an edge device ... but of course others may have different ideas/requirments.

3. I am guessin that is something that could be an addon project anyways.... perhaps using Microsoft Orleans or something similar.

4. Not really a big requirement of us, we just use TPUs sometimes for training for the cost savings, but CUDA is good enough for us.

For adding ONNX export i am guessing this would require exposing additional libtorch endpoints and wrapping them in Torchsharp
am i right ( torchscript is not really useable for us anyways ) however it might be ( have not done a code dive into pytorch internals yet from my understaind that the transform goes model -> tracing -> torchscript -> conversion -> onnx so torchscript would be a base requirement ?

I started prototyping some ideas around a higher level addon library with a similar goal as pytorch lightning here:
TorchSharpFire

especially in regards to a modular data pipeline that supports multithreading and composable transformations.

[NiklasGustafsson]

perhaps using Microsoft Orleans or something similar.

Adding some folks from the Microsoft side: @luisquintanilla, @JakeRadMSFT, @michaelgsharp, @tarekgh.

The issue isn't so much about which distributed framework to rely on (although I like the suggestion :-)) as it is about how to tool it in a way that makes it straightforward to go from an in-memory setup to a distributed one with the least amount of code changes.

ONNX export would require us to analyze the code in a model and figure out what the computation graph looks like, if there is a graph to be formed in the first place. As long as your model is linear (no branches), that is fairly straightforward. For example, if the model just uses 'Sequential,' then that's easy, we just have to map Pytorch operator to ONNX operators and transfer the weights.

However, for models that are not strictly linear (ResNet, for example), it gets more complicated. We need to perform data flow analysis on the forward() function and figure out what the computation graph looks like. That requires something like a Roslyn extension, as far as I can tell. It's not out of the question, of course, but we'd do that in the context of ML.NET / Model Builder integration, since it's getting into tooling, rather than as a standalone feature of TorchSharp.

[BernhardGlueck]

I had thought that the whole tracing functionality is already provided by libtorch ? Or is it implemented on the python side ?

Regardless i think a roslyn analyzer would not help there as it would fail to capture dynamic behaviour.... I am guessing we would need a way to record all the operatons that were invoked dynamically with some kind of tracing. ( as the current one in python works that way as well )

Maybe a stupid idea, but if we generate proxy implementations of all the baseline operators in torchsharp ( or add tracing hooks to the existing source code ) via IL manipulation that could make everything nicely traceable ... and would work similiarly to the current pytorch jit tracing export ....

[NiklasGustafsson]

Sounds like a great project for someone (hint, hint...) to prototype!

[BernhardGlueck]

I am guessing intercepting all the method calls would be feasible, in an addon library, however determining WHAT to intercept would be quite complicated. Most operators at least in the NN namespace derive from nn.Module as far as i can see, and since you standardized the forward method overloads, intercepting those should not be too hard.

However for all the other torch operators we would need to handle that individually in some sort of spec that determines which methods are of interest...

[NiklasGustafsson]

I think a simple way to prototype something is to add derived types for a representative set of torch.nn modules in a new namespace, something like torch.jit.nn, and then manually instrument the constructors and forward() methods in appropriate ways to support tracing. The tracing API can be something as simple as registering a <string,object> dictionary of the parameters for the module plus the final weights.

Then, the prototype tracing just means using torch.jit.nn instead of using torch.nn during tracing.

[NiklasGustafsson]

I'm not suggesting this should be the eventual UX, just for prototyping purposes.

[NiklasGustafsson]

determining WHAT to intercept would be quite complicated. Most operators at least in the NN namespace derive from nn.Module as far as i can see, and since you standardized the forward method overloads, intercepting those should not be too hard.

However for all the other torch operators we would need to handle that individually in some sort of spec that determines which methods are of interest...

I agree -- that will be a little complicated, but if we start with just the standard modules and get the mechanics of tracing in place, then we can move on to the operators, like torch.add(), that are needed but are not modules.