LSTM seq length cast node is causing error while removing unused nodes

[buddhapuneeth]

_remove_unused_nodes() is failing for my model. Unfortunately, I can't share you the model.

assert in_ == '' or in_ in input_dict

(https://github.com/onnx/keras-onnx/blob/master/keras2onnx/topology.py#L230)

Above check is failing me. My values are:

>in_
'masking_1/Any_1:0'
>in_ in input_dict
False

This check was part of the recent change #466

you can see the 'masking_1/Any_1:0' was input of 'cast' node which was build at:
https://github.com/onnx/keras-onnx/blob/master/keras2onnx/ke2onnx/lstm.py#L291

keras-onnx/keras2onnx/ke2onnx/simplernn.py

Line 150 in a75bfa5

def build_sequence_lengths(scope, operator, container):

I am assuming 'masking_1/Any_1:0' to be part of 'input_dict' in _remove_unused_nodes() but is not the case.

Shouldn't we add cast input here into container.add_initializer()?
Env: Keras 2.3.1, keras-onnx master (as of May12th 2020), Py 36

[wenbingl]

It looks the cast node lost its precedent node, you can comment out _remove_unused_nodes and let nodes = container.nodes and generate the model to see what's the root cause.

[buddhapuneeth]

@wenbingl Thanks for your reply.
I already tweaked the code in exactly the way you mentioned. The above screenshot is from the resultant model itself. Without making that change, conversion is failing.
Also, with that tweak, we still get error at optimization but conversion will pass.

  File "../lib/python3.6/site-packages/onnxconverter_common/optimizer.py", line 306, in build_from_onnx
    assert var_ == '' or var_ in inputs
AssertionError

I made below fix to make the conversion work without any error. As sequence length is also an input, I added it into container input list.

Please verify is it correct or not:

container.add_input(operator.input_masks[0])

after the line
https://github.com/onnx/keras-onnx/blob/master/keras2onnx/ke2onnx/simplernn.py#L163

[wenbingl]

In that case, the root cause is the mask/Any_1:0 lost the its connection. I think there is masking layer prior to LSTM layer in the keras model. For some reason, the masking layer output wasn't converted correctly.

The model converted with your tweak may need an extra input of mask tensor to on the inference.

[buddhapuneeth]

@wenbingl yes you are correct, my approach will cause issue while inference and in my case as well there is a masking layer before LSTM in Keras model (.hdf5).

Is it something I need to debug around convert_keras_masking()?
https://github.com/onnx/keras-onnx/blob/master/keras2onnx/ke2onnx/main.py#L90

[wenbingl]

debug_mode=True, then see if "masking_1/Any_1:0" existing in the parsing model log. and If yes, why this layer wasn't converted correctly.

[buddhapuneeth]

@wenbingl your inputs helped me to debug the issue.
Just want to quickly check based on one of the comment in the code (https://github.com/onnx/keras-onnx/blob/master/keras2onnx/_parser_1x.py#L94)

# input mask in a shared model is not supported yet, why is it needed?

Looks like there is no support for shared input mask and in my case, I have shared input mask as below sample:

there is difference between output_mask of masking layer and input_mask of LSTM layers.
Any suggestion for handling this scenario?

[wenbingl]

The short answer is,
can you flat the model? i.e. don't use a sub model in another model, or move the masking layer out of the sub model.

It's the limitation of the current converter, not sure if we can solve it easily.

[buddhapuneeth]

@wenbingl Thanks a lot!