.get_output crashes due with version error

[BillmanH]

I have a notebook that crashes when on 'get_output'.

best_run, fitted_model = run.get_output()
print(best_run)
print(fitted_model)

The error I get:

WARNING - The version of the SDK does not match the version the model was trained on.
WARNING - The consistency in the result may not be guaranteed.
WARNING - Package:azureml-automl-core, training version:1.11.0, current version:1.10.0.post1
Package:azureml-automl-runtime, training version:1.11.0, current version:1.10.0.post1
Package:azureml-core, training version:1.11.0, current version:1.10.0.post1
Package:azureml-dataprep, training version:2.0.2, current version:1.10.1
Package:azureml-dataset-runtime, training version:1.11.0.post1, current version:1.10.0
Package:azureml-defaults, training version:1.11.0, current version:1.10.0
Package:azureml-explain-model, training version:1.11.0, current version:1.10.0
Package:azureml-interpret, training version:1.11.0, current version:1.10.0
Package:azureml-pipeline-core, training version:1.11.0, current version:1.10.0
Package:azureml-telemetry, training version:1.11.0, current version:1.10.0
Package:azureml-train-automl-client, training version:1.11.0, current version:1.10.0
Package:azureml-train-automl-runtime, training version:1.11.0.post1, current version:1.10.0
WARNING - Please ensure the version of your local conda dependencies match the version on which your model was trained in order to properly retrieve your model.
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-54-a9cb6b13e60b> in <module>
----> 1 best_run, fitted_model = run.get_output()

~\AppData\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\train\automl\run.py in get_output(self, iteration, metric, return_onnx_model, return_split_onnx_model, **kwargs)
    593                     )
    594                 else:
--> 595                     fitted_model = self._download_automl_model(curr_run)
    596 
    597             return curr_run, fitted_model

~\AppData\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\train\automl\run.py in _download_automl_model(self, curr_run)
    822                 import azureml.train.automl.runtime
    823                 with open(self.local_model_path, "rb") as model_file:
--> 824                     fitted_model = pickle.load(model_file)  # type: Optional[Any]
    825                     return fitted_model
    826             except ImportError as e:

~\AppData\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\automl\runtime\featurization\__init__.py in <module>
      6 
      7 # Data transformer
----> 8 from .data_transformer import DataTransformer, TransformerAndMapper
      9 
     10 

~\AppData\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\automl\runtime\featurization\data_transformer.py in <module>
     52 from azureml.automl.runtime._engineered_feature_names import (_FeatureTransformers, _GenerateEngineeredFeatureNames,
     53                                                               _Transformer, _TransformerParamsHelper)
---> 54 from ..featurizer.transformer import (AutoMLTransformer, CategoricalFeaturizers, DateTimeFeaturesTransformer,
     55                                       featurization_utilities, GenericFeaturizers, get_ngram_len, TextFeaturizers)
     56 from ..stats_computation import PreprocessingStatistics as _PreprocessingStatistics

~\AppData\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\automl\runtime\featurizer\transformer\__init__.py in <module>
     26 
     27 # Timeseries
---> 28 from .timeseries import TimeSeriesTransformer, TimeSeriesPipelineType, NumericalizeTransformer, \
     29     MissingDummiesTransformer, LaggingTransformer
     30 

~\AppData\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\automl\runtime\featurizer\transformer\timeseries\__init__.py in <module>
     63     is_datetime_like, ALLOWED_TIME_COLUMN_TYPES, type_is_numeric, type_is_one_of
     64 
---> 65 from azureml.automl.core.shared.rolling_origin_validator import RollingOriginValidator
     66 
     67 

ImportError: cannot import name 'RollingOriginValidator' from 'azureml.automl.core.shared.rolling_origin_validator' (...\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\automl\core\shared\rolling_origin_validator.py)

However, I ran this code all in one notebook. So the env that I trained the model is the same that I ran the cell above. The run completes successfully. I can't share the exact notebook, but I could probably mask a couple of things and attach that. The notebook isn't doing anything other than a standard AUTOML run. No special processing or features.

I'm not changing environments during the run. Either this error message is wrong, or the SDK is missing some critical feature.

AzureML version: 1.10.0

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: a0f73dc3-6e33-e99d-de65-c8c3d86cdbf7
• Version Independent ID: 4df3e0e3-4d59-eae6-1646-09c873f8fcea
• Content: azureml.train.automl.run.AutoMLRun class - Azure Machine Learning Python
• Content Source: AzureML-Docset/stable/docs-ref-autogen/azureml-train-automl-client/azureml.train.automl.run.AutoMLRun.yml
• Service: machine-learning
• Sub-service: core
• GitHub Login: @DebFro
• Microsoft Alias: debfro

[pbartos]

Please share AutoMLConfig and compute_target setup.

[BillmanH]

Sure.

automl_config = AutoMLConfig(
    task="classification",
    training_data=training_data,
    label_column_name="SUCCESS",
    experiment_timeout_minutes=60,
    model_explainability=True,
    primary_metric="accuracy",
    compute_target=compute_target,
)

run = experiment.submit(config=automl_config, tags=tags)

and the compute target:

compute_name = "mycompute-vm"

compute_target = ComputeTarget(workspace=ws, name=compute_name)
compute_target

[pbartos]

SDK packages on your remote target are different than on your local machine. Usually, automl experiment sent information about packages version from local env to remote target, but this can be overridden by conda/pip settings.
Find these overrides or reinstall your local sdk version to 1.11.0 and restart the notebook.

[BillmanH]

So you are saying that the SKD will pick a version that is different that the local machine? good to know. If this is a case could you make the error message reply with the version that the cloud used? That way I can follow along. Otherwise I would have to guess. Just having it say that it's out of sync doesn't help me solve the problem.

[BillmanH]

I'm updating now. I'll post if this solves the problem.

[BillmanH]

I used pip install --upgrade azureml-sdk per the docs.
And it updated to 1.12.0. I assumed that this was the stable version as package managers tend to hold back versions that aren't stable.

Running again I got a slightly different error:

ImportError: cannot import name 'RollingOriginValidator' from 'azureml.automl.core.shared.rolling_origin_validator' (...\AppData\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\automl\core\shared\rolling_origin_validator.py)

I'm going to try to install the specific version above (1.11.0), but future viewers should be aware that this solution will likely not work in the future.

[BillmanH]

1.11.0 didn't work either. Same error.

ImportError: cannot import name 'RollingOriginValidator' from 'azureml.automl.core.shared.rolling_origin_validator' (~\AppData\Local\Continuum\anaconda3\envs\azureml\lib\site-packages\azureml\automl\core\shared\rolling_origin_validator.py)

[BillmanH]

I was actually able to do this by using the pip installer for the specific extra attachments
pip install --upgrade azureml-sdk[explain,automl]
That updated my notebook version to 1.12.0.
That seems to have updated the RollingOriginValidator and the notebook was able to complete.

However, there is a larger issue that the AML version in the cloud no longer matches the local version when created. It used to. This means that future users will need to anticipate which SDK version is being used in each run. This also means that they will have to worry that previously trained models can become unusable due to version expiration.

For future users if you get a similar error you cant just update the azureml-sdk, you need to update the underlying packages as well.