How to do schema/data migrations in ElasticSearch

[nelsonic]

We need to write a tutorial on doing Data Migrations when Schema are changed.
https://en.wikipedia.org/wiki/Data_migration

[missinglink]

Elasticsearch isn't set up to do this sort of thing unfortunately, changing the mapping for a field will require a full re-index of that data.

I'm not sure why they chose this design pattern as you can elect to store:true the data verbatim as it's sent to the server, so presumably this _source data could be used to reindex the field after-the-fact.

disclaimer: this may not be true any more as of version 2, (recently released, many things have changed)

migrations may also be discouraged for performance reasons, each segment is immutable, these segments are written once per refresh (default:1s) and then merged in to larger immutable segments every merge operation. (as per leveldb etc.)

elasticsearch gets significant benefits from the linux file-system cache when using these immutable segments (which is why the recommendation is to set 50% RAM for the HEAP and 50% for the fs cache), I can only imagine the performance impact of doing a field migration on a live db 💥

ref: https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html

[missinglink]

you can actually update the settings for a _type without reindexing it but it will require you to temporarily close and then re-open the index. AFAIK this is not possible for the mapping and errors will be thrown.

ref: https://github.com/pelias/schema/blob/master/scripts/update_settings.js

[nelsonic]

@missinglink yeah, we had to re-index a (reasonably) large dataset today after changing a field from an array to an map and it was painful. I'm considering writing an ETL tool to help simplify the job...

[missinglink]

I am on my 4th iteration of a js streaming bulk indexer for ES, but I put it on hold while other stuff took priority. You can find the partially completed repo here with notes about what functionality it should exhibit.

that, coupled with a readStream would do the job nicely, maybe we could work together on finishing that when the time comes?