Skip to content

Some contents causes a fatal error when importing databases into PostgreSQL #224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dpordomingo opened this issue Mar 22, 2018 · 2 comments
Labels
bug Something isn't working

Comments

@dpordomingo
Copy link
Contributor

dpordomingo commented Mar 22, 2018

When a database that is being imported into a PostgreSQL internal database contains weird content, the whole import process fails.

This database error-weird-content.db.zip contains a FilePair that cannot be imported to a PostgreSQL database.

INFO[2018-03-22 17:04:55] request started                               http_method=POST http_proto=HTTP/1.1 http_scheme=http remote_addr="127.0.0.1:53496" uri="http://cat.sourced.tech:8080/api/experiments/1/file-pairs" user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36"
INFO[2018-03-22 17:04:55] Failed to insert row
error: pq: invalid byte sequence for encoding "UTF8": 0xce 0xc4
  http_method=POST http_proto=HTTP/1.1 http_scheme=http remote_addr="127.0.0.1:53496" uri="http://cat.sourced.tech:8080/api/experiments/1/file-pairs" user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36"
ERRO[2018-03-22 17:04:55] pq: Could not complete operation in a failed transaction  http_method=POST http_proto=HTTP/1.1 http_scheme=http remote_addr="127.0.0.1:53496" uri="http://cat.sourced.tech:8080/api/experiments/1/file-pairs" user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36"

That FilePair was found inside the 2000 FilePairs database provided by ML, and it was properly imported into the internal SQlite database on Staging as id#1635

@dpordomingo dpordomingo added this to the PostgreSQL support milestone Mar 22, 2018
@dpordomingo dpordomingo added the bug Something isn't working label Mar 22, 2018
@bzz
Copy link
Contributor

bzz commented Mar 23, 2018

Interesting, did you have a chance to look a bit deeper into this?
I.e does this mean that some content in SQLite we have is already not UTF-8 encoded, or does it gets messed up on the application side?

@dpordomingo
Copy link
Contributor Author

I didn't spend more time digging into this problem.
I only tried to find the way to reproduce it and confirm which FilePair could serve us as an example.
In this case, the file in the zip can be seen in Staging if you go to the pair with id#1635

Should I try to confirm if the content in SQLite is the same as the provided by ML, or if was it ruined when imported also in the internal SQLite DB?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants