Feature update python (#368)

* Update unit_tests.yaml

change python version from 3.10 to 3.12, using existing requirements.txt

* Update requirements.txt

update versions that were installed for Python 3.12 conda environment

* Update unit_tests.yaml

check out the feature_update_python branch of METcalcpy to incorporate the  Python 3.12 changes

* Update requirements.txt

pyyaml should be 6.0.2

* Updating to Python 3.12 use "assert mock_foo.called" rather than "assert mock_foo.called_once_with(...)"
Refer to this: python/cpython#100690

* Update numpy to 2.2.3 to be consistent with METcalcpy and METplotpy

* Update python version

* Change ?=3.12 to =3.12 for Python version

* set minimum python version to 3.11

* Revert back to mimimum python of 3.10.4

* fix missing "

* update escape sequence in lines 432 and 436 to be compatible with Python 3.12

* in pandas to_datetime, replace errors='ignore' to errors='raise' to address the FutureWarning

* Addres FutureWarning: Downcasting behavior in the replace method.  Opt-in to the future behavior and explicitly as specified in StackOverflow: https://stackoverflow.com/questions/79226735/pandas-replace-and-downcasting-deprecation-since-version-2-2-0

* update package versions to be consistent with those specified in the requirements.txt file

* In addressing the FutureWarning, inadvertantly left the request for dtype in command.

* use the feature_update_python branch instead of the develop branch to use the correct package versions

* Explicitly cast the fill value to an int to satisfy the pandas 2.2 expectation of an int dtype

* convert the OBS_LEAD andt FCST_LEAD series to int64 to make all dtypes consistent

* set dtypes to OBS_LEAD and FCST_LEAD columns to facilitate merging

* set dtype of the FCST_LEAD and OBS_LEAD series earlier, to fix error when using pandas merge

* Revert to original code

* Revert the OBS_LEAD column to match original code.  Replicate same method for setting the FCST_LEAD column to int

* set the FCST_LEAD and OBS_LEAD columns as strings, as the fill values in the constants.py file are supposed to be strings

* Explicitly change the type of the fcst_lead and obs_lead columns prior to merging to address the FutureWarning

* Address the ValueError when merging the OBS_LEAD columns between two different dataframes where one type is an int vs object

* After merge, change the OBS_LEAD column to string values to address failing tests

* move dtype reassignment outside of try-except code and catch only exceptions from merge validation issues

* remove try-except and include changing the dtypes for the fcst_lead in both dataframes so they are consistent (and avoiding the ValueError when merging on different dtypes)

* fix indentation error

* fix syntax for verifying merge operation

* Remove the validate option on merging two dataframes

* Explicitly set the dtypes in fcst_column and obs_column for remaining merge operations

* Update requirements.txt

* Update requirements.txt

* Update requirements.txt

* Update nco_requirements.txt

updates to package versions for Python 3.12 migration

* Package versions for Python 3.10

* For testing current code for backwards compatibility with Python 3.10

* Update the names of the runs to facilitate monitoring of GHA's

* Remove the python 3.10 run to isolate issues from running tests

* Updates for Python 3.12

* Python 3.10 version for sonarqube

* Unused/ignored, removing this.

* Testing for Python 3.10

* Update unit_tests.yaml

updated the METcalcpy version from feature_update_python to the develop branch

* Update py310_unit_tests.yaml

update METcalcpy branch to develop

* Update benchmark_pandas2_2.yaml

update METcalcpy branch to develop

* Address another deprecation with m_2d_data.obs_valid

* Removing testing for Python 3.10 due to lack of backwards compatibility in Pandas: pandas-dev/pandas#59845

* Updated for Python 3.12

* Update .readthedocs.yaml

update for Python 3.12

* Update sonarqube.yaml

update METcalcpy branch to the develop branch

Author: bikegeek

.github/workflows/benchmark_pandas2_2.yaml

@@ -32,7 +32,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.10" ]
+        python-version: [ "3.12" ]
 
     steps:
       - uses: actions/checkout@v4
@@ -52,11 +52,11 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          python -m pip install pytest>=7.1.1
+          python -m pip install pytest>=8.3.4
           python -m pip install pytest_benchmark
-          python -m pip install netcdf4==1.6.2
+          python -m pip install netcdf4==1.7.2
           if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
-          python -m pip install pandas==2.2
+          python -m pip install pandas==2.2.3
       
 
       #   Checking the branch name, not necessary but useful when setting things up.

.github/workflows/sonarqube.yaml

@@ -61,10 +61,10 @@ jobs:
         # Disable shallow clones for better analysis
         fetch-depth: 0
 
-    - name: Set up Python 3.10
+    - name: Set up Python 3.12
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
+        python-version: "3.12"
 
     - name: Retrieve METcalcpy repository develop branch
       run: |
@@ -77,8 +77,8 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        python -m pip install pytest>=7.1.1
-        python -m pip install netcdf4==1.6.2
+        python -m pip install pytest>=8.3.4
+        python -m pip install netcdf4==1.7.2
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
         python -m pip install pytest-cov
 

.github/workflows/unit_tests.yaml

@@ -41,7 +41,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ "3.10" ]
+        python-version: [ "3.12" ]
 
     steps:
       - uses: actions/checkout@v4
@@ -50,7 +50,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
 
-      - name: Retrieve METcalcpy repository develop branch
+      - name: Retrieve METcalcpy repository active feature branch
         run: |
           python -m pip install --upgrade pip
           metcalcpy_dir="${RUNNER_WORKSPACE}/METcalcpy"
@@ -62,8 +62,8 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          python -m pip install pytest>=7.1.1
-          python -m pip install netcdf4==1.6.2
+          # python -m pip install pytest>=7.1.1
+          # python -m pip install netcdf4==1.6.2
           if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
           python -m pip install pytest-cov
 

.readthedocs.yaml

@@ -12,7 +12,7 @@ formats: []
 build:
   os: ubuntu-22.04
   tools:
-    python: "3.10"
+    python: "3.12"
 
 # Optionally set the version of Python and requirements required to build your
 # docs

METdbLoad/test/test_met_db_load.py

@@ -455,7 +455,7 @@ def test_empty_files(tmp_path):
 def test_print_version():
     mock_logger = MagicMock()
     print_version(mock_logger)
-    assert mock_logger.info.called_once
+    assert mock_logger.info.called
     assert mock_logger.info.call_args[0][0].startswith("METdbload Version:")
 
     with pytest.raises(SystemExit):

METdbLoad/test/test_tables.py

@@ -267,11 +267,11 @@ def test_something():
     # *** Count mode table rows
     q_line2 = "SELECT table_name, table_rows FROM information_schema.tables " + \
             "WHERE table_schema = '" + DB2 + "' AND " + \
-            "table_name LIKE 'mode\_%';"
+            "table_name LIKE 'mode\\_%';"
 
     q_line3 = "SELECT table_name, table_rows FROM information_schema.tables " + \
             "WHERE table_schema = '" + DB3 + "' AND " + \
-            "table_name LIKE 'mode\_%';"
+            "table_name LIKE 'mode\\_%';"
 
     same = count_rows(q_line2, q_line3)
 
@@ -429,11 +429,11 @@ def test_something():
     # *** Count mtd table rows
     q_line2 = "SELECT table_name, table_rows FROM information_schema.tables " + \
             "WHERE table_schema = '" + DB2 + "' AND " + \
-            "table_name LIKE 'mtd\_%';"
+            "table_name LIKE 'mtd\\_%';"
 
     q_line3 = "SELECT table_name, table_rows FROM information_schema.tables " + \
             "WHERE table_schema = '" + DB3 + "' AND " + \
-            "table_name LIKE 'mtd\_%';"
+            "table_name LIKE 'mtd\\_%';"
 
     same = count_rows(q_line2, q_line3)
 

METdbLoad/ush/read_data_files.py

@@ -549,17 +549,20 @@ def read_data(self, load_flags, load_files, line_types):
                                         mtd_file[CN.FCST_LEAD_HR], unit='sec')
 
                                 # Where fcst_lead was set to zero for math, set it to -9999
+
                                 if mtd_file[CN.FCST_LEAD].eq(0).any():
+                                    mtd_file[CN.FCST_LEAD] = mtd_file[CN.FCST_LEAD].astype(
+                                        str)
                                     mtd_file.loc[mtd_file.fcst_lead ==
-                                                 0, CN.FCST_LEAD] = CN.MV_NOTAV
+                                                 0, CN.FCST_LEAD] = str(CN.MV_NOTAV)
 
                                 # if OBS_LEAD is NA, set it to -9999
                                 if not mtd_file.obs_lead.dtypes == 'int':
+                                    mtd_file[CN.OBS_LEAD] = mtd_file[CN.FCST_LEAD].astype(
+                                        str)
                                     mtd_file.loc[mtd_file.obs_lead ==
-                                                 CN.NOTAV, CN.OBS_LEAD] = CN.MV_NOTAV
-                                    mtd_file[CN.OBS_LEAD] = mtd_file[CN.OBS_LEAD].astype(
-                                        int)
-
+                                                 CN.NOTAV, CN.OBS_LEAD] = str(CN.MV_NOTAV)
+                               
                                 # initially, match line data to the index of the file names
                                 mtd_file[CN.FILE_ROW] = row_num
 
@@ -1534,16 +1537,16 @@ def read_stat(self, filename, hdr_names):
         # convert MET dates to correct date format
         stat_file[CN.FCST_VALID_BEG] = \
             pd.to_datetime(stat_file[CN.FCST_VALID_BEG],
-                           format='%Y%m%d_%H%M%S', errors='ignore')
+                           format='%Y%m%d_%H%M%S', errors='raise')
         stat_file[CN.FCST_VALID_END] = \
             pd.to_datetime(stat_file[CN.FCST_VALID_END],
-                           format='%Y%m%d_%H%M%S', errors='ignore')
+                           format='%Y%m%d_%H%M%S', errors='raise')
         stat_file[CN.OBS_VALID_BEG] = \
             pd.to_datetime(stat_file[CN.OBS_VALID_BEG],
-                           format='%Y%m%d_%H%M%S', errors='ignore')
+                           format='%Y%m%d_%H%M%S', errors='raise')
         stat_file[CN.OBS_VALID_END] = \
             pd.to_datetime(stat_file[CN.OBS_VALID_END],
-                           format='%Y%m%d_%H%M%S', errors='ignore')
+                           format='%Y%m%d_%H%M%S', errors='raise')
         return stat_file
 
     def read_tcst(self, filename, hdr_names):
@@ -1581,10 +1584,10 @@ def read_tcst(self, filename, hdr_names):
         # convert MET dates to correct date format
         stat_file[CN.INIT] = \
             pd.to_datetime(stat_file[CN.INIT],
-                           format='%Y%m%d_%H%M%S', errors='ignore')
+                           format='%Y%m%d_%H%M%S', errors='raise')
         stat_file[CN.VALID] = \
             pd.to_datetime(stat_file[CN.VALID],
-                           format='%Y%m%d_%H%M%S', errors='ignore')
+                           format='%Y%m%d_%H%M%S', errors='raise')
 
         return stat_file
 
@@ -1621,13 +1624,14 @@ def read_mode(self, filename, hdr_names):
         stat_file.columns = hdr_names
 
         # convert MET dates to correct date format
+
         stat_file[CN.FCST_VALID] = \
             pd.to_datetime(stat_file[CN.FCST_VALID],
-                           format='%Y%m%d_%H%M%S', errors='ignore')
+                           format='%Y%m%d_%H%M%S', errors='raise')
         stat_file.loc[stat_file.obs_valid ==
                       CN.NOTAV, CN.OBS_VALID] = CN.MV_NULL
         stat_file[CN.OBS_VALID] = \
             pd.to_datetime(stat_file[CN.OBS_VALID],
-                           format='%Y%m%d_%H%M%S', errors='ignore')
+                           format='%Y%m%d_%H%M%S', errors='raise')
 
         return stat_file

METdbLoad/ush/write_mtd_sql.py

@@ -170,10 +170,17 @@ def write_mtd_data(load_flags, m_2d_data, m_3d_single_data, m_3d_pair_data,
                     m_2d_data.obs_valid = pd.to_datetime(m_2d_data.obs_valid,
                                                          errors='coerce')
                     # put the header ids back into the dataframe
-                    m_2d_data = pd.merge(
-                        left=mtd_headers, right=m_2d_data, on=CN.MTD_2D_HEADER_KEYS)
-                    m_2d_data.loc[m_2d_data.obs_valid.isnull(),
-                                  CN.OBS_VALID] = CN.MV_NULL
+
+                    # to address changes in pandas 2.2, explicitly convert the dtypes
+                    # for the obs_lead and fcst_lead columns in the dataframes that are being merged
+                    headers_to_convert = [CN.OBS_LEAD, CN.FCST_LEAD]
+                    for cur_header in headers_to_convert:
+                        mtd_headers[cur_header] = mtd_headers[cur_header].astype(str)
+                        m_2d_data[cur_header] = m_2d_data[cur_header].astype(str)
+
+                    m_2d_data = pd.merge(left=mtd_headers, right=m_2d_data, on=CN.MTD_2D_HEADER_KEYS)
+                    obs_valid = m_2d_data.obs_valid.astype(str)
+                    m_2d_data.loc[obs_valid.isnull(), CN.OBS_VALID] = CN.MV_NULL
 
                     # create defaults for flags
                     m_2d_data[CN.SIMPLE_FLAG] = 1
@@ -212,6 +219,14 @@ def write_mtd_data(load_flags, m_2d_data, m_3d_single_data, m_3d_pair_data,
                                                                 errors='coerce')
 
                     # put the header ids back into the dataframe
+
+                    # to address changes in pandas 2.2, explicitly convert the dtypes
+                    # for the obs_lead and fcst_lead columns in the dataframes that are being merged
+                    headers_to_convert = [CN.OBS_LEAD, CN.FCST_LEAD]
+                    for cur_header in headers_to_convert:
+                        mtd_headers[cur_header] = mtd_headers[cur_header].astype(str)
+                        m_3d_single_data[cur_header] = m_3d_single_data[cur_header].astype(str)
+
                     m_3d_single_data = pd.merge(left=mtd_headers, right=m_3d_single_data,
                                                 on=CN.MTD_HEADER_KEYS)
                     m_3d_single_data.loc[m_3d_single_data.obs_valid.isnull(
@@ -254,6 +269,13 @@ def write_mtd_data(load_flags, m_2d_data, m_3d_single_data, m_3d_pair_data,
                                                               errors='coerce')
 
                     # put the header ids back into the dataframe
+                    # to address changes in pandas 2.2, explicitly convert the dtypes
+                    # for the obs_lead and fcst_lead columns in the dataframes that are being merged
+                    headers_to_convert = [CN.OBS_LEAD, CN.FCST_LEAD]
+                    for cur_header in headers_to_convert:
+                        mtd_headers[cur_header] = mtd_headers[cur_header].astype(str)
+                        m_3d_pair_data[cur_header] = m_3d_pair_data[cur_header].astype(str)
+
                     m_3d_pair_data = pd.merge(left=mtd_headers, right=m_3d_pair_data,
                                               on=CN.MTD_HEADER_KEYS)
                     m_3d_pair_data.loc[m_3d_pair_data.obs_valid.isnull(

METdbLoad/ush/write_stat_sql.py

@@ -143,6 +143,7 @@ def write_stat_data(load_flags, stat_data, tmp_dir, sql_cur, local_infile, logge
                                 str(len(line_data.index)))
 
                     # change all Not Available values to METviewer not available (-9999)
+                    pd.set_option('future.no_silent_downcasting', True)
                     line_data = line_data.replace(CN.NOTAV, CN.MV_NOTAV)
 
                     # Only variable length lines have a line_data_id

docs/Users_Guide/installation.rst

@@ -32,13 +32,16 @@ typing the command in the CLI and/or make the corresponding edit to
 Python Requirements
 -------------------
 
-**Python 3.10.4+** - Python 3.10.4 or higher must be installed. 
+**Python 3.12+** - Python 3.12 or higher must be installed. 
+
+**NOTE** Due to the lack of backwards compatibility in pandas 2.x, this code no longer works with Python 3.10
 
 The requirements below come directly from the **requirements.txt** file 
 at the top level of the repository.
 
 .. literalinclude:: ../../requirements.txt
 
+
 Installation
 ============
 

nco_requirements.txt

@@ -1,5 +1,5 @@
-   lxml>=4.9.1
-   pandas>=1.5.2
-   pyyaml>=6.0
-   pytest>=7.2.1
-   numpy>=1.24.2
+   lxml>=5.3.0
+   pandas>=2.2.3
+   pyyaml>=6.0.2
+   pytest>=8.3.4
+   numpy>=2.2.2

py310_requirements.txt

@@ -0,0 +1,12 @@
+lxml==4.9.1
+numpy==1.24.2
+pandas==1.5.2
+pip==23.3
+PyMySQL==1.1.1
+pytest==7.2.1
+python-dateutil==2.8.2
+PyYAML==6.0
+xarray>=2023.1.0
+
+
+

pyproject.toml

@@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "metdataio"
 #dynamic = ["dependencies"]
-version = "3.0.0-rc1-dev"
+version = "3.1.0-beta2-dev"
 description = "METplus component that loads data into a database"
 authors = [
     {name = "METplus"},
@@ -53,4 +53,4 @@ source = [
     ]
 omit = [
     "METreformat/test/*"
-]
+]

requirements.txt

@@ -1,12 +1,12 @@
-lxml==4.9.1
-numpy==1.24.2
-pandas==1.5.2
-pip==23.3
+lxml==5.3.0
+numpy==2.2.2
+pandas==2.2.3
+pip==25.0
 PyMySQL==1.1.1
-pytest==7.2.1
-python-dateutil==2.8.2
-PyYAML==6.0
-xarray>=2023.1.0
+pytest==8.3.4
+python-dateutil==2.9.0.post0
+PyYAML==6.0.2
+xarray>=2025.1.2