Update validations in core

[itsarijitray]

This PR aims to achieve 2 things:-

• Expose validateSchema() (We have use cases for the same. Refer to STRATCONN-5861 and STRATCONN-4327)
• Allow segment internal fields (__segment_internal_sync_mode, __segment_internal_matching_key) for mapping validations: In braze-cloud's upsertCatalogItem we have a field which is required for a certain syncMode (upsert). To run this validation, we need to allow internal field validation while not exposing the internal fields in the payload (this block already takes care of that. Refer ajv docs).

Testing

Sample Mapping Config:

Here it can be seen that item object is required only case of "__segment_internal_sync_mode": "upsert"

Sample API test when item object is passed

{
  "specversion": "2.0",
  "id": "1234567890",
  "source": "fake-source",
  "destination": "650bdf1a62fb34ef0a8058e1",
  "type": "com.segment.integrations.actions-braze-cloud.track",
  "time": "Thu Oct 07 2024 15:29:00 GMT+0530 (India Standard Time)",
  "deadline": "Thu Oct 07 2024 18:29:01 GMT+0530 (India Standard Time)",
  "context": {},
  "transaction": {},
  "settings": {
    "subscriptions": [
      {
        "subscribe": "type = \"track\"",
        "enabled": true,
        "partnerAction": "upsertCatalogItem",
        "mapping": {
          "__segment_internal_sync_mode": "upsert",
          "catalog_name": "cars",
          "item_id": {
            "@path": "$.properties.id"
          },
          "item": {
            "name": {
              "@path": "$.properties.name"
            },
            "launch_date": {
              "@path": "$.properties.launch_date"
            },
            "inception_date": {
              "@path": "$.properties.inception_date"
            },
            "price": {
              "@path": "$.properties.price"
            },
            "discontinued": {
              "@path": "$.properties.discontinued"
            },
            "manufacturer": {
              "@path": "$.properties.company"
            }
          },
          "enable_batching": true
        }
      }
    ],
    "api_key": "XXXXXXXXXXXXXXXX",
    "endpoint": "https://rest.iad-03.braze.com"
  },
  "features": [
    "cloudevent-spec-v02-all"
  ],
  "data": [
    {
      "type": "track",
      "channel": "server",
      "event": "Update Catalog",
      "properties": {
        "id": "car003",
        "name": "Beetle",
        "company": "Volkswagen",
        "price": 9000,
        "discontinued": true,
        "inception_date": "1938-01-28",
        "launch_date": "1940-01-27"
      }
    },
    {
      "type": "track",
      "event": "Update Catalog",
      "channel": "server",
      "properties": {
        "id": "car002",
        "name": "Mustang 2005",
        "company": "Ford",
        "price": 55999.5,
        "in_stock": false,
        "discontinued": false,
        "inception_date": "1964-04-17T07:00:00Z"
      }
    },
    {
      "type": "track",
            "channel": "server",

      "properties": {
        "id": "car056",
        "discontinued": false,

      },
      "event": "Update Catalog"
    },
    {
      "type": "track",
      "event": "Update Catalog",
      "channel": "server",
      "properties": {
        "id": "car001",
        "name": "Model S",
        "company": "Tesla",
        "price": 4,
        "discontinued": false,
        "launch_date": "2012-06-22T04:00:00Z"
      }
    },
    {
      "type": "track",
      "channel": "server",
      "properties": {
        "id": "car056",
        "launch_date": "2012-06-22T04:00:00Z"

      },
      "event": "Update Catalog"
    }
  ]
}

As item object is present, API doesn't throw any errors

{
  "id": "1234567890",
  "source": "fake-source",
  "specversion": "2.0",
  "type": "com.segment.event.ack",
  "time": "2025-05-30T07:07:38.270Z",
  "status": 207,
  "trace": {
    "name": "invoke",
    "time": "2025-05-30T07:07:34.971Z",
    "duration": "3.299s",
    "logs": [],
    "spans": [
      {
        "name": "http_request",
        "time": "2025-05-30T07:07:37.487Z",
        "duration": "0.778s",
        "logs": [
          {
            "time": "2025-05-30T07:07:37.593Z",
            "key": "INFO",
            "value": "connected to 104.17.242.75:443"
          },
          {
            "time": "2025-05-30T07:07:38.265Z",
            "key": "INFO",
            "value": "PUT https://rest.iad-03.braze.com/catalogs/cars/items/ (422B) - 202 Accepted (46B)"
          }
        ],
        "spans": [],
        "exchange": {
          "time": "2025-05-30T07:07:37.487Z",
          "duration": "0.778s",
          "request": {
            "protocol": "HTTP/1.1",
            "method": "PUT",
            "url": "https://rest.iad-03.braze.com/catalogs/cars/items/",
            "header": {
              "Authorization": "Bearer ea447ac6-8ab5-462f-ad53-1ffe1b03230c",
              "User-Agent": "Segment (Actions)",
              "Content-Type": "application/json",
              "Accept": "*/*",
              "Content-Length": "422",
              "Accept-Encoding": "gzip,deflate",
              "Connection": "close",
              "Host": "rest.iad-03.braze.com"
            },
            "body": "eyJpdGVtcyI6W3siaWQiOiJjYXIwMDMiLCJuYW1lIjoiQmVldGxlIiwibGF1bmNoX2RhdGUiOiIxOTQwLTAxLTI3IiwiaW5jZXB0aW9uX2RhdGUiOiIxOTM4LTAxLTI4IiwicHJpY2UiOjkwMDAsImRpc2NvbnRpbnVlZCI6dHJ1ZSwibWFudWZhY3R1cmVyIjoiVm9sa3N3YWdlbiJ9LHsiaWQiOiJjYXIwMDIiLCJuYW1lIjoiTXVzdGFuZyAyMDA1IiwiaW5jZXB0aW9uX2RhdGUiOiIxOTY0LTA0LTE3VDA3OjAwOjAwWiIsInByaWNlIjo1NTk5OS41LCJkaXNjb250aW51ZWQiOmZhbHNlLCJtYW51ZmFjdHVyZXIiOiJGb3JkIn0seyJpZCI6ImNhcjAwMSIsIm5hbWUiOiJNb2RlbCBTIiwibGF1bmNoX2RhdGUiOiIyMDEyLTA2LTIyVDA0OjAwOjAwWiIsInByaWNlIjo0LCJkaXNjb250aW51ZWQiOmZhbHNlLCJtYW51ZmFjdHVyZXIiOiJUZXNsYSJ9XX0="
          },
          "response": {
            "protocol": "HTTP/1.1",
            "statusCode": 202,
            "statusText": "Accepted",
            "header": {
              "Date": "Fri, 30 May 2025 07:07:38 GMT",
              "Content-Type": "application/json",
              "Transfer-Encoding": "chunked",
              "Connection": "close",
              "Cf-Ray": "947c65491a5c599c-DEL",
              "Cf-Cache-Status": "DYNAMIC",
              "Cache-Control": "no-cache",
              "Content-Encoding": "gzip",
              "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
              "Vary": "Origin,Accept-Encoding",
              "X-Ratelimit-Limit": "16000",
              "X-Ratelimit-Remaining": "15999",
              "X-Ratelimit-Reset": "1748588880",
              "X-Request-Id": "ea576aa9-c1fa-463d-932b-4df49c6fe7d5",
              "X-Runtime": "0.059321",
              "Server": "cloudflare"
            },
            "body": "H4sIADpZOWgAA6pWyk0tLk5MT1WyUiouTU4GcpRqAQAAAP//AwDGVSbjFQAAAA=="
          }
        }
      }
    ]
  },
  "responses": [
    {
      "status": 200,
      "events": [
        {
          "indices": [
            0,
            1,
            2,
            3,
            4
          ],
          "data": {
            "message": "success"
          }
        }
      ]
    }
  ]
}

Sample API test when item object is not passed

{
  "specversion": "2.0",
  "id": "1234567890",
  "source": "fake-source",
  "destination": "650bdf1a62fb34ef0a8058e1",
  "type": "com.segment.integrations.actions-braze-cloud.track",
  "time": "Thu Oct 07 2024 15:29:00 GMT+0530 (India Standard Time)",
  "deadline": "Thu Oct 07 2024 18:29:01 GMT+0530 (India Standard Time)",
  "context": {},
  "transaction": {},
  "settings": {
    "subscriptions": [
      {
        "subscribe": "type = \"track\"",
        "enabled": true,
        "partnerAction": "upsertCatalogItem",
        "mapping": {
          "__segment_internal_sync_mode": "upsert",
          "catalog_name": "cars",
          "item_id": {
            "@path": "$.properties.id"
          },
          "enable_batching": true
        }
      }
    ],
    "api_key": "XXXXXXXXXXXXXXX",
    "endpoint": "https://rest.iad-03.braze.com"
  },
  "features": [
    "cloudevent-spec-v02-all"
  ],
  "data": [
    {
      "type": "track",
      "channel": "server",
      "event": "Update Catalog",
      "properties": {
        "id": "car003",
        "name": "Beetle",
        "company": "Volkswagen",
        "price": 9000,
        "discontinued": true,
        "inception_date": "1938-01-28",
        "launch_date": "1940-01-27"
      }
    },
    {
      "type": "track",
      "event": "Update Catalog",
      "channel": "server",
      "properties": {
        "id": "car002",
        "name": "Mustang 2005",
        "company": "Ford",
        "price": 55999.5,
        "in_stock": false,
        "discontinued": false,
        "inception_date": "1964-04-17T07:00:00Z"
      }
    },
    {
      "type": "track",
            "channel": "server",

      "properties": {
        "id": "car056",
                "discontinued": false

      },
      "event": "Update Catalog"
    },
    {
      "type": "track",
      "event": "Update Catalog",
      "channel": "server",
      "properties": {
        "id": "car001",
        "name": "Model S",
        "company": "Tesla",
        "price": 4,
        "discontinued": false,
        "launch_date": "2012-06-22T04:00:00Z"
      }
    },
    {
      "type": "track",
      "channel": "server",
      "properties": {
        "id": "car056",
                "launch_date": "2012-06-22T04:00:00Z"

      },
      "event": "Update Catalog"
    }
  ]
}

API throws error

{
  "id": "1234567890",
  "source": "fake-source",
  "specversion": "2.0",
  "type": "com.segment.event.ack",
  "time": "2025-05-30T07:20:12.504Z",
  "status": 207,
  "trace": {
    "name": "invoke",
    "time": "2025-05-30T07:20:10.450Z",
    "duration": "2.054s",
    "logs": [],
    "spans": []
  },
  "responses": [
    {
      "status": 400,
      "events": [
        {
          "indices": [],
          "errortype": "PAYLOAD_VALIDATION_FAILED",
          "errormessage": "The root value is missing the required field 'item'. The root value must match \"then\" schema.",
          "errorreporter": "INTEGRATIONS"
        }
      ]
    }
  ]
}

• Added unit tests for new functionality
• Tested end-to-end using the local server
• [If destination is already live] Tested for backward compatibility of destination. Note: New required fields are a breaking change.
• [Segmenters] Tested in the staging environment
• [Segmenters] [If applicable for this change] Tested for regression with Hadron.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.67%. Comparing base (0451a3d) to head (986c7b1).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2934      +/-   ##
==========================================
+ Coverage   78.07%   78.67%   +0.59%     
==========================================
  Files        1064     1117      +53     
  Lines       19612    22412    +2800     
  Branches     3767     4524     +757     
==========================================
+ Hits        15313    17633    +2320     
- Misses       3015     3489     +474     
- Partials     1284     1290       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[varadarajan-tw]

This makes me a little worried. Why is adding the internal field in sent required here?

[itsarijitray]

It is because of this condition. Basically the tests don't have schema definition, thats why the internal fields are not removed by validateSchema(). To fix this, We can either add schema to the test, or remove the internal fields here and then add the internal fields back before validateSchema()