apache_beam + beam integration not sending exceptions in GCP #4203
Comments
|
Hey @gshoultz42 ! What you also can do:
Hope this helps |
|
I just fed your question into our bot in our discord server, and it has these suggestions: https://discord.com/channels/621778831602221064/1354782809020960779/1354782809020960779 (its not super helpful I have to admit, but it is a nice experiment) |
|
Sentry Init sentry_sdk.init(
dsn=sentry_dsn,
environment=env,
integrations=[BeamIntegration()],
)Exception call class ParseEvent(beam.DoFn):
def process(self, element: bytes, *args, **kwargs):
try:
event_json = json.loads(element)
event_name = event_json["message"]["payload"]["eventName"]
model = get_model(event_name)
event = model.model_validate_json(element)
yield event
except (ValidationError, KeyError, JSONDecodeError, TypeError) as error:
logging.exception("Error parsing event: %r", element)
error_output = {
"error": repr(error),
"payload": (
event_json
if not isinstance(error, JSONDecodeError)
else element.decode()
),
}
yield TaggedOutput("parse_errors", json.dumps(error_output).encode()) |
|
Thanks for the follow up! The line logging.exception("Error parsing event: %r", element)should send an error if the logging integration is enabled (which the sentry SDK enables by default.) If you want to have your Because you handle the exception yourself, it never bubbles up for Sentry to capture it. You could also add a simple |
|
Wanted to provide an update. I am still looking into this. I added debug to sentry but did not get the log. Working with our admin to make sure the logging is setup correctly in our GCP |
|
I see this on the main job but I don't see any logs from the worker logs where the exception is happening [sentry] DEBUG: Sending envelope [envelope with 1 items (error)] project:xxx host:o58632.ingest.us.sentry.io |
|
Hey @gshoultz42 thanks for the logs. From the log you posted I see: There is also a debug from urllib3 that the post request succeeded: So everything looks normal and the error should end up in Sentry. As far as I can tell, the SDK is not the problem. You can have a look at https://sentry.io/stats/ to see if received errors have been dropped. (Errors can be dropped by "inbound data filters" (that you can configure) or if you are over quota, or by rate limitting if you send tons of errors at the same time) |
|
That error did go up but it wasn't the exception that was being fired to be captured.
I can see the log entry for the expected exception but I don't see that exception in sentry.
This is what I have noticed.
- The main job exceptions can be caught.
- Any exceptions coming from a worker is not caught.
- Sentry logs are not making it to the logs in gcp. I can only view the logs from my console as I deploy.
Is there a way to force sentry to use a different logger?
|
|
The line You can do a In general Sentry is only capturing unhandled errors from your application. If you catch them and handle them, they will not be sent to Sentry. |
|
Current code block that is not working except (ValidationError, KeyError, JSONDecodeError, TypeError) as error:
with sentry_sdk.push_scope() as scope:
# Add any additional context
scope.set_extra("exception", str(error))
sentry_sdk.capture_exception(error)
exc_info = exc_info_from_error(error)
exceptions = exceptions_from_error_tuple(exc_info)
sentry_sdk.capture_event(
{
"message": str(error),
"level": "error",
"exception": {"values": exceptions},
}
)
logging.exception("Error parsing event: %r", element)
error_output = {
"error": repr(error),
"payload": (
event_json
if not isinstance(error, JSONDecodeError)
else element.decode()
),
}I should have posted this sooner. I made some changes mentioned in the discord channel as well. |
|
I did the following actions as well.
The sentry apache_beam demo is not a streaming dataflow job and I am curious if it is something related to that or if it is related to the GCP included parts of the apache_beam sdk |
|
Hi @gshoultz42, to send the exception to Sentry, all you should need is the except (ValidationError, KeyError, JSONDecodeError, TypeError) as error:
+ sentry_sdk.capture_exception(error)
- with sentry_sdk.push_scope() as scope:
- # Add any additional context
- scope.set_extra("exception", str(error))
- sentry_sdk.capture_exception(error)
- exc_info = exc_info_from_error(error)
- exceptions = exceptions_from_error_tuple(exc_info)
- sentry_sdk.capture_event(
- {
- "message": str(error),
- "level": "error",
- "exception": {"values": exceptions},
- }
- )
logging.exception("Error parsing event: %r", element)
error_output = {
"error": repr(error),
"payload": (
event_json
if not isinstance(error, JSONDecodeError)
else element.decode()
),
}Note that the default issue view in Sentry only shows issues with medium or high issue priority. Manually captured exceptions typically have low priority, and would be filtered out by the default filter. You can see the filter in the "Issues" page search bar:
To see the manually-captured exception, you need to remove the issue priority filter. |
|
That was what I had in the beginning but I updated to the code just in case and I got the same result. |
@gshoultz42 I understand, that must be quite frustrating. In any case, you should use the simplified code snippet I sent you with only the But, just to confirm, which Python SDK version were you using just now? And, you seeing any other log outputs from sources other than the Sentry-Python SDK? Also, is the error event still not showing up in Sentry? It would be super helpful if you could also provide the full logs from your latest attempt, and also the code you are running (especially the |
How do you use Sentry?
Sentry Saas (sentry.io)
Version
2.23.1
Steps to Reproduce
Expected Result
Error exception is sent to Sentry
Actual Result
Sentry does not get an exception.
The text was updated successfully, but these errors were encountered: