Basic data structures and working flows for java TensorFlow servers and clients.
This jar provides data structures and methods to convert between Java types and data structures used by a TensorFlow SavedModelBundle class for inference, as well as model loading, dropping, multiple models requesting schedules, GRPC client interface, etc.
This jar can be used to build a TensorFlow server which is able to reading and serving a number of exported TensorFlow models, operates model resources, and also maintains model updating schedules. An example server project can be accessed here.
A class of a simple TensorFlow GRPC client, including data converting methods, is also provided for convenience.
<dependency>
<groupId>com.eroelf</groupId>
<artifactId>tfserver-core</artifactId>
<version>${version}</version>
</dependency>
The data passed to a SavedModelBundle TensorFlow session should always be multi-dimensional arrays. Model Name, model version, model signature, and names of the inputs are also required according to the TensorFlow model export protocol. All these information can be carried by a Sample interface, in which a Map whose keys are the inputs names and values are instances of ArrayWrapper interfaces is used to carry the inputs data. The Sample interface also provides methods to convert between Java data and multi-dimensional arrays which can be accepted by a SavedModelBundle TensorFlow session.
With the help of those data structures and tools described below, one can easily build a TensorFlow server in Java. A server project using this jar and provides both an HTTP Json and a GRPC interface can be found here.
The Model class is the basic unit to run a TensorFlow exported model. A ModelHandler instance maintains extra configurations of a Model. The ModelGroup class is used to maintain a number of ModelHandler objects which are different versions under a same model name. Finally, the ModelPool class is designed to maintain a bunch of ModelGroup objects.
The run method of each class is the key function to run a model.
The data structures described above provide both the Java List version (with a suffix "4J") and the ProtoBuf version (with a suffix "4Pb"). The ProtoBuf data structures are defined for GRPC framework. The definitions of the ProtoBuf and GRPC data structure and protocols can be found in the "datastream.proto" file in this jar. It is highly recommended to use the ProtoBuf version in production environments.
Please note that, most of the data structures generated by the datastream.proto file have the same or similar class names with the data structures described above, but in a different generated package: com.eroelf.tfserver.datastream.
The definition of the request and the response data structure is the RequestInfo message and the Response message, respectively.
A snapshot:
// in datastream.proto
// ...
// The request parameters:
// Additional parameters
message Param
{
string trace=1;// Using for tracing
}
// ...
// A tensor data
message DataArray
{
int32 type_num=1;// The type number of the TensorFlow DataType.
repeated uint32 shape=1;// Tensor shape. Informs the server to convert the one dimensional "data" to the corresponding high dimensional tensor.
google.protobuf.Any data=2;// Tensor data, a one dimensional array, Any means it can accept any data type. It is the user's responsibility to ensure the correct data type.
}
// Sample data
message Sample
{
string model_name=1;// The model name.
string model_version=2;// The model version.
string signature_name=3;// The signature.
map<string, DataArray> inputs=4;// The inputs map
}
// Request information
message RequestInfo
{
Param param=1;
Sample sample=2;
}
// ...
// The response:
message Status
{
sint32 code=1;// The status code
string des=2;// The status description
}
message Response
{
Status status=1;// The response status
map<string, DataArray> data=2;// The results, in other word, the outputs of the model requested.
}
// ...The definitions of the request and response data structure convert tools can be found in ArrayWrapper4J.java and Sample4J.java.
A snapshot:
// The request parameters:
// in ArrayWrapper4J.java
// A tensor data
public class ArrayWrapper4J<T> implements ArrayWrapper
{
private DataType type;// The TensorFlow DataType.
private int[] shape;// Tensor shape. Informs the server to convert the one dimensional "data" to the corresponding high dimensional tensor.
private Object data;// Tensor data, converted from a Java *List* according to the shape value.
}
/**************** separator ****************/
// in Sample4J.java
// Sample data
public class Sample4J implements Sample
{
public String modelName;// The model name.
public String modelVersion;// The model version.
public String signatureName;// The signature.
public Map<String, ArrayWrapper4J<?>> inputs;// The inputs map
// ...
}This class provides methods to build a GRPC client for requesting models running in a TensorFlow GRPC server. Methods for converting between Java data and the ProtoBuf and GRPC data structures are also provided for convenience.
Used in servers. Thrown to indicate that a request parameter has broken the working flow.
- UnregisteredModelException: error code 10. Thrown to indicate that the specified model has not been registered to the TensorFlow server.
- UnregisteredVersionException: error code 11. Thrown to indicate that the specified model has not had the passed version.
- UnregisteredSignatureException: error code 12. Thrown to indicate that the specified model has not had the passed signature.
- UnregisteredInputException: error code 13. Thrown to indicate that the specified model has not had the passed input.
- UnsupportedDataTypeException: error code 20. Thrown to indicate that the passed data type has not been supported.
- DataTypeMismatchException: error code 21. Thrown to indicate that the passed data type has not matched the recorded one.
- ShapeMismatchExceptioin: error code 30. Thrown to indicate that a wrong shape has been passed.
Used in clients. Thrown to indicate that an invalid response has been received from the server.
- ZHONG Weikun
This project is released under the Apache license 2.0.
Copyright 2018 ZHONG Weikun.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.