TypeVarTuple Transformations Before Unpack

[llchan]

It would be useful to be able to apply type transformations to TypeVarTuple's types. Some motivating examples:

 Ts = TypeVarTuple('Ts')

 class Example(Generic[*Ts]):
     def __init__(self, *types: *type[Ts]) -> None:
         self.types = types

     def create_defaults(self) -> tuple[*Ts]:
         return tuple(t() for t in self.types)

     def create_lists(self) -> tuple[*list[Ts]]:
         return tuple([t()] for t in self.types)

     def create_callables(self) -> tuple[*Callable[[], Ts]]:
         return tuple(lambda: t() for t in self.types)

 e = Example(int, str)

 assert_type(e.create_defaults(), tuple[int, str])
 assert_type(e.create_lists(), tuple[list[int], list[str]])
 assert_type(e.create_callables(), tuple[Callable[[], int], Callable[[], str]])

At a high level, everything between the * and the TypeVarTuple it expands would be applied per type in the tuple before it's unpacked.

There are some decisions to be made about how the expansion is done if there are multiple TypeVarTuples in the expand expression, but I think it can be supported and I think the intuitive thing would be to zip up all the TypeVarTuples. For example,
tuple[*tuple[Ts, Us]] with Ts = (int, str) and Us = (float, bytes) would be tuple[tuple[int, float], tuple[str, bytes]], with the implementation enforcing equal-length TypeVarTuples.

If there's a workaround for this using the existing logic, let me know. My current use case involves keeping a reference to the runtime types, so *type[Ts] is what I'm looking for at the moment.

[pradeep90]

@llchan Yes, we plan to add a follow-up PEP to PEP 646 with support for such mapping (hopefully, later this year). We cut it from the original PEP because the PEP was already pretty complicated.

[llchan]

Great to hear. If/when there's an official tracking issue or link to a discussion it would be helpful to include a reference here (and feel free to close this if it's superceded).

[uriyyo]

Hi @pradeep90,

It would be great if it also will be possible to "unwrap" TypeVarTuple.

For instance:

from typing import TypeVarTuple, Generic, TypeVar
from dataclasses import dataclass

T = TypeVar("T")


@dataclass(frozen=True)
class Wrapper(Generic[T]):
    wrapped: T


Ts = TypeVarTuple("Ts")


def unwrap(*objs: *Wrapper[Ts]) -> tuple[*Ts]:
    return tuple(obj.wrapped for obj in objs)

I would love to hear your opinion regarding this feature.

[maflAT]

I ran into this use case today, trying to annotate Qt for Python's Slot() decorator.
It requires the types of the decorated methods' arguments as parameters.
https://stackoverflow.com/questions/76234660/type-hinting-instances-vs-types-with-variadic-generics

Has there been any development regarding this topic since it was opened?

[yakMM]

This would be great, and I'm interested for a workaround for *type[Ts] if any.

[Kroppeb]

It would also be nice to be able to this to TypedDicts and ParamSpecs (and ParamSpecArgs and ParamSpecKwargs)

[Gobot1234]

As an aside would there be a way to map a tuple[T1, T2, T3] -> tuple[U1, U2, U3], in effect just preserving the shape but not the types themselves. Use case (pretending bounds exist)

def buy_items[*AppShopItemTs: AppShopItem](self, *items: *AppShopItemTs) -> ...:  # not sure what the return should look like
    """Buy the ``items`` from the in-app store.
    Returns: The items in the user's inventory, bought from the store"""

def buy_items[*AppShopItemTs: AppShopItem](self, *items: *AppShopItemTs) -> tuple[*(Item for _ in range(len(AppShopItemTs)))]:

is a cute idea and I think I have seen generator expressions as an idea for a shorthand to Map come up a few times

[insilications]

We definitely need this kind of typing.Map-like feature

[randolf-scholz]

Probably duplicate of #1273, specialized to TypeVarTuple.

This would indeed be very nice, another use-case: column-oriented tables like pyarrow.Table or pandas.DataFrame.

class Table[*Dtypes]:
     column_names: tuple[*(str for dtype in *Dtypes)]
     columns: tuple[*(Series[dtype] for dtype in *Dtypes)]

table: Table[int, str] = Table({"foo" : [1,2,3]}, "bar": ["x", "y", "z"])
table.column_names  #  -> tuple[str, str]
table.columns  # -> tuple[Series[int], Series[str]]

EDIT: Updated using the proposed comprehension syntax by @zen-xu for easier readability

[phdowling]

Not sure if this is exactly the same structure as others in this thread, but here's possibly another use-case of this sort of thing: I have a function my_func accepts that accepts a list of callables some_callables that output different types, and the my_func's output will match exactly one of those types:

def my_func[*Ts](
    some_callables: list[
        callable[
            [],
            OneOf[*Ts]  # or however one would express this - maybe "Map", or even "T in Ts" ?
    ]]
) -> Union[*Ts]:
    return random.choice(some_callables)()

def f() -> int:
    return 1

def g() -> str:
    return ""

x: int | str = my_func([f, g])  # the type hint here should be inferrable

I think that list[callable[..., Union[*Ts]]] would be the wrong type for the outputs themselves (they each output one of the types, not a Union), thus the fictional OneOf (or Map, or whatever).

[randolf-scholz]

Here is another related example:

def prod_fn[*Us, *Vs](*funcs: ???) -> Callable[[*Us], tuple[*Vs]]:
    r"""Cartesian Product of Functions.

    It is assumed every function takes a single positional argument.
    """

    def __prod_fn(*args: *Us) -> tuple[*Vs]:
        """Argument is a tuple with the input for each function."""
        return tuple(f(arg) for f, arg in zip(funcs, args))

    return __prod_fn

Here, funcs is the variable length tuple[Callable[[U₁], V₁]], Callable[[U₂], V₂]], ..., Callable[[Uₙ], Vₙ]]]. Possibly syntax:

• *funcs: *Map[(Us, Vs), Callable[[Us], Vs]]] (suggested by David in this discussion)
• *funcs: *tuple[Callable[[U], V]] for U, V in zip(Us, Vs)] (comprehension syntax suggested by @zen-xu)

[Kroppeb]

*funcs: *Map[(U, V), Callable[[U], V]]]

Mmh, I find that notation a bit confusing. The left U refers to the TypeVarTuple and the right U refers to the item in the U tuple?

[randolf-scholz]

Mmh, I find that notation a bit confusing. The left U refers to the TypeVarTuple and the right U refers to the item in the U tuple?

Hm, I guess notationally something like: *funcs: *Map[Func, Us, Vs], with the type alias type Func[U,V] = Callable[[U], V] would be better, with a typing.Map that behaves like builtins.map, but for generic types.

Notably, such a construct is also required to even type hint builtins.map properly. If we look at the current stubs in typeshed, it's only a Generic Iterator[S] with manual @overloads for up to 6 iterables. It probably should be type hinted be something along the lines of:

class map[F: Callable[[*Ts], S], *Ts]:
    @overload
    def __new__(cls, func: F, /) -> Never: ...  # need at least 1 iterable.
    @overload
    def __new__(cls, func: F, /, *iterables: *Map[Iterable, Ts]) -> Self: ...
    def __iter__(self) -> Self: ...
    def __next__(self) -> S: ...

[tanrbobanr]

Are there any plans on adding this feature? It was mentioned that they were planning to release a follow-up PEP later in 2022, but it has been two years. Has this been abandoned or forgotten?

[erictraut]

There are no concrete plans currently.

Keep in mind that it took nearly two years for mypy to implement support for the existing TypeVarTuple functionality defined in PEP 646. Pyre and pytype still don't support it.

If you're interested in proposing extensions to TypeVarTuple and would like to champion a new PEP, a good starting point is to create a discussion in the Python typing forum here.

[tanrbobanr]

Alright, thanks for the quick response! I am interested in championing a new PEP for this, although am unsure if I have enough experience to do so. I will have to look into it.

[zen-xu]

If TypeVarTuple supports list comprehension

T = TypeVar('T')
Ts = TypeVarTuple('Ts')

class Example(Generic[*Ts]):
   def __init__(self, *types: *type[Ts]) -> None:
       self.types = types

   def create_defaults(self) -> tuple[T for T in Ts]:
       return tuple(t() for t in self.types)

   def create_lists(self) -> tuple[list[T] for T in Ts]:
       return tuple([t()] for t in self.types)

   def create_callables(self) -> tuple[Callable[[], T] for T in Ts]:
       return tuple(lambda: t() for t in self.types)

e = Example(int, str)

assert_type(e.create_defaults(), tuple[int, str])
assert_type(e.create_lists(), tuple[list[int], list[str]])
assert_type(e.create_callables(), tuple[Callable[[], int], Callable[[], str]])

In addition, it also supports for type guards.

@overload
def create(self) -> tuple[ContainerA[T] for T in Ts if issubclass(T, A)]: ...

@overload
def create(self) -> tuple[ContainerB[T] for T in Ts if issubclass(T, B)]: ...

[ahmed-mahran]

For people interested in *type[Ts] workarounds, you may check https://github.com/ahmed-mahran/funpy/blob/main/typeargs.py.
There is also a PoC implementation of variadic generics transformations at https://github.com/ahmed-mahran/funpy/blob/main/mapped.py.

[jorenham]

@ahmed-mahran It looks like your also trying to implement higher-kinded typing (HKT) with your examples, see #548 .

---

This made me realize that this issue is actually about variadic HKT.

[bharel]

SO questions requesting that feature

https://stackoverflow.com/q/73200382/1658617
https://stackoverflow.com/q/78954770/1658617

[fzimmermann89]

Additional SO question:
https://stackoverflow.com/questions/78013162/typehint-function-args-tupleargs-with-constraint-on-the-args