Elasticsearch Migrate v1.7
This is a scripting process I used to move Elasticsearch clusters from one location to another.
Dump Command from Bucket to JSON File
elasticdump --input=http://user:pass@bar-10858-0.es.eshost.com:10858/bucket --output=/home/username/elasticsearch-backup/bucket-$(date +\%Y-\%m-\%d)-am.json --type=data --bulk=true --limit=1000
Full Clone
This Crontab runs automated backup from the local backup directory to an S3 bucket
# 1 5 * * * cd /path/elasticsearch-backup/; ls -tr | head -n -12 | xargs --no-run-if-empty rm # OLD method of deleting old files, now s3 handles it
## Every 10 minutes check for json files that need to be compressed that are at least 8 minutes old
*/10 * * * * cd /path/elasticsearch-backup && find ./ -type f -mmin +8 -name "*.json" -exec gzip {} \;
45 15 * * * aws s3 sync /path/elasticsearch-backup/ s3://acme-backup/elasticsearch-backup && rm /path/elasticsearch-backup/*
This script runs a full clone from one destination to another including Settings, Mappings and Data and provides also Aliases the new index with the original Alias. It also deletes older versions of the same repos, but keeps the most recent just in case.
#!/bin/bash
TIME=$(date +%s);
DBFILE="/path/scripts/devclone/acme_devclone.db"
DELDBFILE="/path/scripts/devclone/acme_devclone.db.del"
PRD="http://foo_user:????@bar-10877-0.es.eshost.com:10877"
DEV="http://bar_user:????@bar-10855-0.es.eshost.com:10855"
SRC_ALIAS="acme_prod"
DST_ALIAS="acme_dev"
NEW_INDEX="${DST_ALIAS}_${TIME}"
SRC="$PRD"
DST="$DEV"
# Get old index
OLD_INDEX=$(cat "$DBFILE")
if [ "$OLD_INDEX" == "" ]; then
echo "Old index not found"
exit 1
fi
# Get index to delete
DEL_INDEX=$(cat "$DELDBFILE")
if [ "$DEL_INDEX" == "" ]; then
echo "Delete index not found"
exit 1
fi
# Get settings
SETTINGS=$(curl -XGET "$SRC/$SRC_ALIAS/_settings" | sed -e 's/{"'$SRC_ALIAS'_.*":{"settings"://' | sed -e 's/}}$//')
MAPPINGS=$(curl -XGET "$SRC/$SRC_ALIAS/_mappings" | sed -e 's/{"'$SRC_ALIAS'_.*":{"mappings"://' | sed -e 's/}}$//')
# Create new Index
echo "Creating new index..."
echo "ALIAS: $DST_ALIAS DEL: $DEL_INDEX OLD: $OLD_INDEX NEW: $NEW_INDEX"
curl -XPUT "$DST/$NEW_INDEX" -d "{ \"settings\": $SETTINGS, \"mappings\": $MAPPINGS }"
# Clone data
echo "Cloning data.."
elasticdump \
--input="${SRC}/${SRC_ALIAS}" \
--output="${DST}/${NEW_INDEX}" \
--type=data --bulk=true --limit=1000 --bulk-use-output-index-name=true
# Re-alias
echo "Aliasing new index.."
curl -XPOST "$DST/_aliases" -d "
{
\"actions\" : [
{ \"add\" : { \"index\" : \"$NEW_INDEX\", \"alias\" : \"$DST_ALIAS\" } },
{ \"remove\" : { \"index\" : \"$OLD_INDEX\", \"alias\" : \"$DST_ALIAS\" } }
]
}"
# Save new index as old
echo "Saving new index $NEW_INDEX as old.."
echo "$NEW_INDEX" > "$DBFILE"
# Save old index for deletion
echo "Saving old index $OLD_INDEX to delete.."
echo "$OLD_INDEX" > "$DELDBFILE"
# Delete the index previously marked for deletion
echo "Deleting index $DEL_INDEX.."
curl -XDELETE "$DST/$DEL_INDEX"
echo "..Done."