TF v2 support for Recurrent/Bidirectional

[cjermain]

@DronistB posted an issue on #413 showing that the new TensorFlow V2 API for Keras does not work with the existing conversion code for Bidirectional RNN layers. This occurs because TensorFlow's new RNN layers are in a different module (tensorflow.python.keras.layers.recurrent_v2). At a minimum, the following code will need to change to support the TensorFlow versions as well as the conventional Keras classes.

keras-onnx/keras2onnx/ke2onnx/bidirectional.py

Lines 25 to 37 in ff17787

	@cvtfunc(shape_infer=_calculate_keras_bidirectional_output_shapes)
	def convert_bidirectional(scope, operator, container):
	op_type = type(operator.raw_operator.forward_layer)
	bidirectional = True
	if op_type == keras.layers.LSTM:
	lstm.convert_keras_lstm(scope, operator, container, bidirectional)
	elif op_type == keras.layers.GRU:
	gru.convert_keras_gru(scope, operator, container, bidirectional)
	elif op_type == keras.layers.SimpleRNN:
	simplernn.convert_keras_simple_rnn(scope, operator, container, bidirectional)
	else:
	raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))

Happy to take a look at resolving this when I have bandwidth.

[jiafatom]

@cjermain , thank you for raising this 👍

[BarzelS]

Thanks @cjermain,
So as I understood the problem is with my tensorflow and keras versions?
With which versions will it work?

Update:

I've tried
TensorFlow 1.13.1
Keras: '2.2.4'
Python: 2.7

And got:
raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))
ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.GRU'>

Another Update:
The conversion worked with:
keras: 2.2.0
tensorflow: 1.13.1
Python: 3.6

But with this warnings:

2020-09-06 11:15:57.313114: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz
2020-09-06 11:15:57.313597: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x2982f10 executing computations on platform Host. Devices:
2020-09-06 11:15:57.313627: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
WARNING:tensorflow:No training configuration found in save file: the model was not compiled. Compile it manually.
The ONNX operator number change on the optimization: 49 -> 21

But when tried it with the ModelImporter.cpp of TensorRT got this:

ModelImporter.cpp:125: bidirectional_3 [LSTM] inputs: [fan_in_adjustment_out0 -> (-1, -1, -1)], [bidirectional_3_W -> (2, 512, 256)], [bidirectional_3_R -> (2, 512, 128)], [bidirectional_3_B -> (2, 1024)], [optional input, not set], [optional input, not set], [optional input, not set], [optional input, not set],
ERROR: builtin_op_importers.cpp:1811 In function importLSTM:
[8] Assertion failed: std::equal(activationAlphas.begin(), activationAlphas.begin() + NUM_ACTIVATIONS, activationAlphas.begin() + NUM_ACTIVATIONS) && "The parser does not currently support cases where activations for the reverse pass of the LSTM do not match the forward pass."

[cjermain]

Based on the GRU import path (tensorflow.python.keras.layers.recurrent_v2.GRU) it looks like you are actually using a TF 2.0 compatible version of the GRU -- this was not actually present in TF 1.13.1 (looking at the source code).

https://github.com/tensorflow/tensorflow/blob/v1.14.0/tensorflow/python/keras/layers/recurrent_v2.py

I'm looking into the changes needed to support these classes -- recurrent_v2.GRU breaks tests for me on 1.15.

Have you checked over your model with Netron? It is worth verifying that both the original model and the ONNX model look right.

[cjermain]

For me, using TF 2.2.0 resolves the issues with recurrent_v2.GRU -- it seems they are handled correctly for TF > 2.0. @DronistB , are you able to convert the model using a newer version of TF?

[cjermain]

@DronistB, I made changes in #610 which will allow TF 1.14 and 1.15 to handle the v2 capability. I would suggest upgrading TensorFlow past 2.0 if possible, since that is likely faster -- otherwise let me know if the fix works for you on the TF versions mentioned.

[BarzelS]

Thanks @cjermain,

1. Tried again, now performed training using:

Tensorflow: 2.3.0
Keras. 2.4.3
Python: 3.6

Then:

git clone https://github.com/cjermain/keras-onnx.git
cd keras-onnx
git checkout recurrent_v2_compatibility 
sudo python setup.py install

Then used this script to convert the hdf5 file I got from the training:

import os
os.environ['TF_KERAS'] = '2'
import keras2onnx
from keras.models import load_model
import tensorflow as tf
import argparse

print("Tensorflow version:" + str(tf.__version__))
# arguments
parser = argparse.ArgumentParser()
parser.add_argument('input_file', help='Input file path')
args = parser.parse_args()

# load keras model
from tensorflow.keras.models import load_model
tf.keras.backend.set_learning_phase(0)
tf.keras.backend.clear_session()
model = load_model(args.input_file)

# convert to onnx model
onnx_model = keras2onnx.convert_keras(model, model.name, target_opset=10)

temp_model_file = os.path.splitext(args.input_file)[0] + '_converted.onnx'
keras2onnx.save_model(onnx_model, temp_model_file)

But got this error:

  File "/usr/local/lib/python3.6/dist-packages/keras2onnx/ke2onnx/bidirectional.py", line 37, in convert_bidirectional
    raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))
ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.LSTM'>

2. Similar error when I changed the from "Bidirectional(LSTM" to "Bidirectional(GRU"

(Same versions as above)

  File "/usr/local/lib/python3.6/dist-packages/keras2onnx/ke2onnx/bidirectional.py", line 37, in convert_bidirectional
    raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))
ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.GRU'>

3. Tried training using:

Tensorflow: 1.13.1
Keras. 2.2.4
Python: 2.3.6

And converting using:

Tensorflow: 2.3.0
Keras. 2.4.3
Python: 3.6

But again:

  File "/usr/local/lib/python3.6/dist-packages/keras2onnx/ke2onnx/bidirectional.py", line 37, in convert_bidirectional
    raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))
ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.GRU'>

4. Tried also to convert with
   Tensorflow: 1.13.1
   Keras. 2.2.0
   Python: 3.6.9

But got this error:

site-packages/tensorflow/python/keras/utils/generic_utils.py", line 166, in class_and_config_for_serialized_keras_object
raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown layer: Functional

Thanks

[BarzelS]

Thanks @cjermain,

1. Tried again, now performed training using:

Tensorflow: 2.3.0
Keras. 2.4.3
Python: 3.6

Then:

git clone https://github.com/cjermain/keras-onnx.git
cd keras-onnx
git checkout recurrent_v2_compatibility 
sudo python setup.py install

Then used this script to convert the hdf5 file I got from the training:

import os
os.environ['TF_KERAS'] = '2'
import keras2onnx
from keras.models import load_model
import tensorflow as tf
import argparse

print("Tensorflow version:" + str(tf.__version__))
# arguments
parser = argparse.ArgumentParser()
parser.add_argument('input_file', help='Input file path')
args = parser.parse_args()

# load keras model
from tensorflow.keras.models import load_model
tf.keras.backend.set_learning_phase(0)
tf.keras.backend.clear_session()
model = load_model(args.input_file)

# convert to onnx model
onnx_model = keras2onnx.convert_keras(model, model.name, target_opset=10)

temp_model_file = os.path.splitext(args.input_file)[0] + '_converted.onnx'
keras2onnx.save_model(onnx_model, temp_model_file)

But got this error:

  File "/usr/local/lib/python3.6/dist-packages/keras2onnx/ke2onnx/bidirectional.py", line 37, in convert_bidirectional
    raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))
ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.LSTM'>

1. Similar error when I changed the from "Bidirectional(LSTM" to "Bidirectional(GRU"

(Same versions as above)

  File "/usr/local/lib/python3.6/dist-packages/keras2onnx/ke2onnx/bidirectional.py", line 37, in convert_bidirectional
    raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))
ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.GRU'>

1. Tried training using:

Tensorflow: 1.13.1
Keras. 2.2.4
Python: 2.3.6

And converting using:

Tensorflow: 2.3.0
Keras. 2.4.3
Python: 3.6

But again:

  File "/usr/local/lib/python3.6/dist-packages/keras2onnx/ke2onnx/bidirectional.py", line 37, in convert_bidirectional
    raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))
ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.GRU'>

1. Tried also to convert with
   Tensorflow: 1.13.1
   Keras. 2.2.0
   Python: 3.6.9

But got this error:

site-packages/tensorflow/python/keras/utils/generic_utils.py", line 166, in class_and_config_for_serialized_keras_object
raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown layer: Functional

Thanks

Hi @cjermain, any updates for the above comment?

[volgaone]

Hello,
I am running into this very issue, it seems. I am trying to export a BiLSTM model to onnx, but I get the following error:

~\Anaconda3\envs\py3.7\lib\site-packages\keras2onnx\ke2onnx\bidirectional.py in convert_bidirectional(scope, operator, container)
     35         simplernn.convert_keras_simple_rnn(scope, operator, container, bidirectional)
     36     else:
---> 37         raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))

ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.LSTM'>

I have attached the model which I am trying to load:
bilstm.zip

And here's the code I am using:

import onnx
import keras2onnx
from keras.models import load_model
from tensorflow.keras.models import load_model

in_model = "../../data/models/master/bilstm.h5"
out_model = "../../data/models/master/bilstm.onnx"

model = load_model(in_model)
onnx_model = keras2onnx.convert_keras(model, model.name)
onnx.save_model(onnx_model, out_model)

I would really appreciate any help. Happy to help test pull requests!
Thanks

[davidnarganes]

Hi,

Any updates on this issue? I've the same error with:

keras==2.4.3
tensorflow==2.3.1
keras2onnx==1.7.0
onnxmltools==1.7.0

Thanks

[cjermain]

File "/usr/local/lib/python3.6/dist-packages/keras2onnx/ke2onnx/bidirectional.py", line 37, in convert_bidirectional
raise ValueError('Unsupported class for Bidirectional layer: {}'.format(op_type))
ValueError: Unsupported class for Bidirectional layer: <class 'tensorflow.python.keras.layers.recurrent_v2.LSTM'>

@DronistB: With my changes, the ValueError exception should occur on line 46 -- line 37 indicates you are not actually running with those changes. Make sure to install the package in a virtual or conda environment, and avoid using sudo to install Python packages. For example, here is one strategy that might help:

$ conda create --name recurrent-v2 python=3.7 tensorflow
$ conda activate recurrent-v2
(recurrent-v2)$ pip install -U onnxruntime
(recurrent-v2)$ pip install -U git+https://github.com/microsoft/onnxconverter-common
(recurrent-v2)$ pip install -U git+https://github.com/cjermain/keras-onnx@recurrent_v2_compatibility

The last line will give you my recurrent_v2_compatability branch of keras-onnx for #610, which has the fix. The main package does not have the fix yet.

@volgaone: I have run the above and successfully converted your model. I attached the original and ONNX files in the following ZIP: bilstm.zip

@davidnarganes: Once I get the builds passing for #610, we should be able to get this into the official package. Until then, you can use the above instructions to use the PR branch to bypass the issue.