feat: Adding support for Native Python feature transformations for On Demand Feature Views

[franciscojavierarceo]

What this PR does / why we need it:

This pull request adds support for native Python transformation. Historically, On Demand Feature Views only supported pandas. We introduce support for python in this Pull Request.

The resulting behavior should look something like this:

from feast import Field, RequestSource
from feast.types import Float64, Int64
import pandas as pd

# Define a request data source which encodes features / information only
# available at request time (e.g. part of the user initiated HTTP request)
input_request = RequestSource(
    name="vals_to_add",
    schema=[
        Field(name='val_to_add', dtype=Int64),
        Field(name='val_to_add_2', dtype=Int64)
    ]
)

# Use the input data and feature view features to create new features
@on_demand_feature_view(
   sources=[
       driver_hourly_stats_view,
       input_request
   ],
   schema=[
     Field(name='conv_rate_plus_val1', dtype=Float64),
     Field(name='conv_rate_plus_val2', dtype=Float64)
   ],
   mode="python"
)
def transformed_conv_rate(features: Dict[str, Any]) -> pd.DataFrame:
    output = {}
    output['conv_rate_plus_val1'] = 2 + features['conv_rate'] + features['val_to_add']
    output['conv_rate_plus_val2'] = 2 + features['conv_rate'] + features['val_to_add_2']
    return output

Note this has nice performance gains when serving particularly for single rows.

Continuation of:

• feat: Updating protos to separate transformation #4018
• feat: Rename OnDemandTransformations to Transformations #4038

Which issue(s) this PR fixes:

Fixes #

[HaoXuAI]

Overall looks good to me!
Awesome work and this definitely should be featured in the next release and document/post

[HaoXuAI]

I guess I haven't followed the code in details, just want to know what if the input is something like a text string to transform , or a purely number to calculate, instead of dictionary?

[franciscojavierarceo]

return should be Dict[str, Any]

[HaoXuAI]

The string 'pandas' can be replaced by a enum to avoid typo, but we don't have many "modes" so I think it's good for now

[franciscojavierarceo]

yeah i was going to refactor later and do some clean up to centralize

[tokoko]

this kinda looks like it's breaking substrait, but I guess it's fine. seems like online path isn't tested for substrait in ci, I'll add tests and a fix (if necessary) for this later.

[tokoko]

LGTM

[franciscojavierarceo]

@HaoXuAI see here as example of usage.

[HaoXuAI]

LGTM！

[tokoko]

@franciscojavierarceo sorry for the late find. Can you look at this test examples? The signatures are correct, but actual transformation seems to assume that features_dict["feature1"] is an int, but it should be a list of ints, right?

[franciscojavierarceo]

whoops let me update the test case tomorrow but it should be a singleton, the type hint is wrong. I adjusted this late and missed this one.

[tokoko]

ah, got it. If it should be a singleton, then also double-check what is actually passed from odfv class as well. I may be wrong, but I think it's passing a list.

[franciscojavierarceo]

I'll add another test when using get_online_features for an ODFV to confirm it more explicitly. I do have a test below confirming the correct behavior for self.feature_transformation.transform() returning the expected result.

[tokoko]

I'm planning to add an explicit get_online_features test for substrait as well. Just testing transform is clearly not enough.

[franciscojavierarceo]

here's the fix for the type: #4054