-
Notifications
You must be signed in to change notification settings - Fork 18.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
caffe.io.array_to_datum fails with some dtypes #2391
Comments
Thanks for reporting the issue. I've changed the title to be a little more specific. We should probably just change |
@longjon - Thanks. I like your fix better. I suppose I got too fussy about maintaining variable type, which may not be truly important for this function. I'd be happy to "PR" this, but I don't know what it means to "PR" something.... |
@tubaybb321 PR stands for pull request (docs). |
As recommended by @longjon, this will allow `caffe.io.array_to_datum` to handle, for example, numpy.float32 arrays. It might be worth noting that `datum.float_data` is stored as protobuf type 2, which is float32, as opposed to protobuf type 1, which is float64. It is a little unintuitive that caffe currently requires data to be passed in as float64 but then writes float32 to LMDB. To demonstrate this: ```python datum = caffe.io.array_to_datum(np.array([[[0.9]]])) caffe.io.datum_to_array(datum) # array([[[ 0.9]]]) datum_str = datum.SerializeToString() new_datum = caffe.proto.caffe_pb2.Datum() new_datum.ParseFromString(datum_str) caffe.io.datum_to_array(new_datum) # array([[[ 0.89999998]]]) ``` This behavior is somewhat hidden because `datum_to_array` returns type float64, even though the data doesn't actually have that resolution if it has been stored as protobuf text anywhere (for example in LMDB). Alternative solutions: * Require and return float32, consistent with the protobuf representation. * Change the protobuf to allow float32 or float64 and update surrounding code to support this.
Hi @longjon, @seanbell, @tubaybb321! Pull request #4182 contains the suggested fix, with a little additional discussion in a commit message there as well. I'll copy in that message for those finding this thread: As recommended by @longjon, [PR #4182] will allow It might be worth noting that datum = caffe.io.array_to_datum(np.array([[[0.9]]]))
caffe.io.datum_to_array(datum)
# array([[[ 0.9]]])
datum_str = datum.SerializeToString()
new_datum = caffe.proto.caffe_pb2.Datum()
new_datum.ParseFromString(datum_str)
caffe.io.datum_to_array(new_datum)
# array([[[ 0.89999998]]]) This behavior is somewhat hidden because Alternative solutions:
|
is this fixed in newer builds? I'm facing the same issue here where I'm stuck with float64 and I run out of memory because of it (cant save the normalized data into a leveldb , I have 32G of RAM by the way!) |
Fixed in #2391 |
convert non-uint8 dtypes to float; refs #2391
As recommended by @longjon, this will allow `caffe.io.array_to_datum` to handle, for example, numpy.float32 arrays. It might be worth noting that `datum.float_data` is stored as protobuf type 2, which is float32, as opposed to protobuf type 1, which is float64. It is a little unintuitive that caffe currently requires data to be passed in as float64 but then writes float32 to LMDB. To demonstrate this: ```python datum = caffe.io.array_to_datum(np.array([[[0.9]]])) caffe.io.datum_to_array(datum) # array([[[ 0.9]]]) datum_str = datum.SerializeToString() new_datum = caffe.proto.caffe_pb2.Datum() new_datum.ParseFromString(datum_str) caffe.io.datum_to_array(new_datum) # array([[[ 0.89999998]]]) ``` This behavior is somewhat hidden because `datum_to_array` returns type float64, even though the data doesn't actually have that resolution if it has been stored as protobuf text anywhere (for example in LMDB). Alternative solutions: * Require and return float32, consistent with the protobuf representation. * Change the protobuf to allow float32 or float64 and update surrounding code to support this.
As recommended by @longjon, this will allow `caffe.io.array_to_datum` to handle, for example, numpy.float32 arrays. It might be worth noting that `datum.float_data` is stored as protobuf type 2, which is float32, as opposed to protobuf type 1, which is float64. It is a little unintuitive that caffe currently requires data to be passed in as float64 but then writes float32 to LMDB. To demonstrate this: ```python datum = caffe.io.array_to_datum(np.array([[[0.9]]])) caffe.io.datum_to_array(datum) # array([[[ 0.9]]]) datum_str = datum.SerializeToString() new_datum = caffe.proto.caffe_pb2.Datum() new_datum.ParseFromString(datum_str) caffe.io.datum_to_array(new_datum) # array([[[ 0.89999998]]]) ``` This behavior is somewhat hidden because `datum_to_array` returns type float64, even though the data doesn't actually have that resolution if it has been stored as protobuf text anywhere (for example in LMDB). Alternative solutions: * Require and return float32, consistent with the protobuf representation. * Change the protobuf to allow float32 or float64 and update surrounding code to support this.
As recommended by @longjon, this will allow `caffe.io.array_to_datum` to handle, for example, numpy.float32 arrays. It might be worth noting that `datum.float_data` is stored as protobuf type 2, which is float32, as opposed to protobuf type 1, which is float64. It is a little unintuitive that caffe currently requires data to be passed in as float64 but then writes float32 to LMDB. To demonstrate this: ```python datum = caffe.io.array_to_datum(np.array([[[0.9]]])) caffe.io.datum_to_array(datum) # array([[[ 0.9]]]) datum_str = datum.SerializeToString() new_datum = caffe.proto.caffe_pb2.Datum() new_datum.ParseFromString(datum_str) caffe.io.datum_to_array(new_datum) # array([[[ 0.89999998]]]) ``` This behavior is somewhat hidden because `datum_to_array` returns type float64, even though the data doesn't actually have that resolution if it has been stored as protobuf text anywhere (for example in LMDB). Alternative solutions: * Require and return float32, consistent with the protobuf representation. * Change the protobuf to allow float32 or float64 and update surrounding code to support this.
As recommended by @longjon, this will allow `caffe.io.array_to_datum` to handle, for example, numpy.float32 arrays. It might be worth noting that `datum.float_data` is stored as protobuf type 2, which is float32, as opposed to protobuf type 1, which is float64. It is a little unintuitive that caffe currently requires data to be passed in as float64 but then writes float32 to LMDB. To demonstrate this: ```python datum = caffe.io.array_to_datum(np.array([[[0.9]]])) caffe.io.datum_to_array(datum) # array([[[ 0.9]]]) datum_str = datum.SerializeToString() new_datum = caffe.proto.caffe_pb2.Datum() new_datum.ParseFromString(datum_str) caffe.io.datum_to_array(new_datum) # array([[[ 0.89999998]]]) ``` This behavior is somewhat hidden because `datum_to_array` returns type float64, even though the data doesn't actually have that resolution if it has been stored as protobuf text anywhere (for example in LMDB). Alternative solutions: * Require and return float32, consistent with the protobuf representation. * Change the protobuf to allow float32 or float64 and update surrounding code to support this.
As recommended by @longjon, this will allow `caffe.io.array_to_datum` to handle, for example, numpy.float32 arrays. It might be worth noting that `datum.float_data` is stored as protobuf type 2, which is float32, as opposed to protobuf type 1, which is float64. It is a little unintuitive that caffe currently requires data to be passed in as float64 but then writes float32 to LMDB. To demonstrate this: ```python datum = caffe.io.array_to_datum(np.array([[[0.9]]])) caffe.io.datum_to_array(datum) # array([[[ 0.9]]]) datum_str = datum.SerializeToString() new_datum = caffe.proto.caffe_pb2.Datum() new_datum.ParseFromString(datum_str) caffe.io.datum_to_array(new_datum) # array([[[ 0.89999998]]]) ``` This behavior is somewhat hidden because `datum_to_array` returns type float64, even though the data doesn't actually have that resolution if it has been stored as protobuf text anywhere (for example in LMDB). Alternative solutions: * Require and return float32, consistent with the protobuf representation. * Change the protobuf to allow float32 or float64 and update surrounding code to support this.
@longjon Thanks your solution worked for me, very simple and helpful. I just save the array as float before sending to
|
I think there may be a problem with caffe.io.array_to_datum. It only seems to convert arrays of type uint8 or float64. I would have expected it to handle arrays of type int32 and float32.
This is what I saw:
The problem, I believe, lies in the use of the extend function to store any non-uint8 data in the float_data field of a Datum object. Although this is very robust for lists, the float_data field for a Datum object is a google.protobuf.internal.containers.RepeatedScalarFieldContainer, which is more type sensitive when using the extend function, as can be seen in the error trace:
The two ways of I see to fix this are:
Either to change protobuf's type checking, so that it allows for numpy.types. I believe this would involve modifying the following section of code (lines 196-208 in type_checkers.py):
Or, to modify array_to_datum, so that when it flattens an array, the returned iterator does the type casting for protobuf:
This gave me the following results, so it appears to be working fine:
There may still be an issue that other than uint8, all ints will be the system's default int type (i.e. either 32 or 64 bit), all unsigned ints will also be the system's default int type and all floats will be the default float type. But, for now, unless there is a better solution, I think it's OK to use this. Am I correct in believing this? Is there a better approach?
Thanks,
Steven
The text was updated successfully, but these errors were encountered: