# Elasticsearch Migrate v1.7 This is a scripting process I used to move Elasticsearch clusters from one location to another. ## Dump Command from Bucket to JSON File ```bash elasticdump --input=http://user:pass@bar-10858-0.es.eshost.com:10858/bucket --output=/home/username/elasticsearch-backup/bucket-$(date +\%Y-\%m-\%d)-am.json --type=data --bulk=true --limit=1000 ``` ## Full Clone This Crontab runs automated backup from the local backup directory to an S3 bucket ```shell # 1 5 * * * cd /path/elasticsearch-backup/; ls -tr | head -n -12 | xargs --no-run-if-empty rm # OLD method of deleting old files, now s3 handles it ## Every 10 minutes check for json files that need to be compressed that are at least 8 minutes old */10 * * * * cd /path/elasticsearch-backup && find ./ -type f -mmin +8 -name "*.json" -exec gzip {} \; 45 15 * * * aws s3 sync /path/elasticsearch-backup/ s3://acme-backup/elasticsearch-backup && rm /path/elasticsearch-backup/* ``` This script runs a full clone from one destination to another including Settings, Mappings and Data and provides also Aliases the new index with the original Alias. It also deletes older versions of the same repos, but keeps the most recent just in case. ```bash #!/bin/bash TIME=$(date +%s); DBFILE="/path/scripts/devclone/acme_devclone.db" DELDBFILE="/path/scripts/devclone/acme_devclone.db.del" PRD="http://foo_user:????@bar-10877-0.es.eshost.com:10877" DEV="http://bar_user:????@bar-10855-0.es.eshost.com:10855" SRC_ALIAS="acme_prod" DST_ALIAS="acme_dev" NEW_INDEX="${DST_ALIAS}_${TIME}" SRC="$PRD" DST="$DEV" # Get old index OLD_INDEX=$(cat "$DBFILE") if [ "$OLD_INDEX" == "" ]; then echo "Old index not found" exit 1 fi # Get index to delete DEL_INDEX=$(cat "$DELDBFILE") if [ "$DEL_INDEX" == "" ]; then echo "Delete index not found" exit 1 fi # Get settings SETTINGS=$(curl -XGET "$SRC/$SRC_ALIAS/_settings" | sed -e 's/{"'$SRC_ALIAS'_.*":{"settings"://' | sed -e 's/}}$//') MAPPINGS=$(curl -XGET "$SRC/$SRC_ALIAS/_mappings" | sed -e 's/{"'$SRC_ALIAS'_.*":{"mappings"://' | sed -e 's/}}$//') # Create new Index echo "Creating new index..." echo "ALIAS: $DST_ALIAS DEL: $DEL_INDEX OLD: $OLD_INDEX NEW: $NEW_INDEX" curl -XPUT "$DST/$NEW_INDEX" -d "{ \"settings\": $SETTINGS, \"mappings\": $MAPPINGS }" # Clone data echo "Cloning data.." elasticdump \ --input="${SRC}/${SRC_ALIAS}" \ --output="${DST}/${NEW_INDEX}" \ --type=data --bulk=true --limit=1000 --bulk-use-output-index-name=true # Re-alias echo "Aliasing new index.." curl -XPOST "$DST/_aliases" -d " { \"actions\" : [ { \"add\" : { \"index\" : \"$NEW_INDEX\", \"alias\" : \"$DST_ALIAS\" } }, { \"remove\" : { \"index\" : \"$OLD_INDEX\", \"alias\" : \"$DST_ALIAS\" } } ] }" # Save new index as old echo "Saving new index $NEW_INDEX as old.." echo "$NEW_INDEX" > "$DBFILE" # Save old index for deletion echo "Saving old index $OLD_INDEX to delete.." echo "$OLD_INDEX" > "$DELDBFILE" # Delete the index previously marked for deletion echo "Deleting index $DEL_INDEX.." curl -XDELETE "$DST/$DEL_INDEX" echo "..Done." ```