Jordan Savant // Software Engineer

Elasticsearch Migrate v1.7

This is a scripting process I used to move Elasticsearch clusters from one location to another.

Dump Command from Bucket to JSON File

elasticdump --input=http://user:pass@bar-10858-0.es.eshost.com:10858/bucket --output=/home/username/elasticsearch-backup/bucket-$(date +\%Y-\%m-\%d)-am.json --type=data --bulk=true --limit=1000

Full Clone

This Crontab runs automated backup from the local backup directory to an S3 bucket

# 1 5 * * * cd /path/elasticsearch-backup/; ls -tr | head -n -12 | xargs --no-run-if-empty rm # OLD method of deleting old files, now s3 handles it
## Every 10 minutes check for json files that need to be compressed that are at least 8 minutes old
*/10 * * * * cd /path/elasticsearch-backup && find ./ -type f -mmin +8 -name "*.json" -exec gzip {} \;
45 15 * * * aws s3 sync /path/elasticsearch-backup/ s3://acme-backup/elasticsearch-backup && rm /path/elasticsearch-backup/*

This script runs a full clone from one destination to another including Settings, Mappings and Data and provides also Aliases the new index with the original Alias. It also deletes older versions of the same repos, but keeps the most recent just in case.

#!/bin/bash

TIME=$(date +%s);

DBFILE="/path/scripts/devclone/acme_devclone.db"
DELDBFILE="/path/scripts/devclone/acme_devclone.db.del"

PRD="http://foo_user:????@bar-10877-0.es.eshost.com:10877"
DEV="http://bar_user:????@bar-10855-0.es.eshost.com:10855"

SRC_ALIAS="acme_prod"
DST_ALIAS="acme_dev"
NEW_INDEX="${DST_ALIAS}_${TIME}"
SRC="$PRD"
DST="$DEV"

# Get old index
OLD_INDEX=$(cat "$DBFILE")
if [ "$OLD_INDEX" == "" ]; then
    echo "Old index not found"
    exit 1
fi

# Get index to delete
DEL_INDEX=$(cat "$DELDBFILE")
if [ "$DEL_INDEX" == "" ]; then
    echo "Delete index not found"
    exit 1
fi

# Get settings
SETTINGS=$(curl -XGET "$SRC/$SRC_ALIAS/_settings" | sed -e 's/{"'$SRC_ALIAS'_.*":{"settings"://' | sed -e 's/}}$//')
MAPPINGS=$(curl -XGET "$SRC/$SRC_ALIAS/_mappings" | sed -e 's/{"'$SRC_ALIAS'_.*":{"mappings"://' | sed -e 's/}}$//')

# Create new Index
echo "Creating new index..."
echo "ALIAS: $DST_ALIAS    DEL: $DEL_INDEX     OLD: $OLD_INDEX      NEW: $NEW_INDEX"
curl -XPUT "$DST/$NEW_INDEX" -d "{ \"settings\": $SETTINGS, \"mappings\": $MAPPINGS }"

# Clone data
echo "Cloning data.."
elasticdump \
    --input="${SRC}/${SRC_ALIAS}" \
    --output="${DST}/${NEW_INDEX}" \
    --type=data --bulk=true --limit=1000 --bulk-use-output-index-name=true

# Re-alias
echo "Aliasing new index.."
curl -XPOST "$DST/_aliases" -d "
{
  \"actions\" : [
    { \"add\" : { \"index\" : \"$NEW_INDEX\", \"alias\" : \"$DST_ALIAS\" } },
    { \"remove\" : { \"index\" : \"$OLD_INDEX\", \"alias\" : \"$DST_ALIAS\" } }
  ]
}"

# Save new index as old
echo "Saving new index $NEW_INDEX as old.."
echo "$NEW_INDEX" > "$DBFILE"

# Save old index for deletion
echo "Saving old index $OLD_INDEX to delete.."
echo "$OLD_INDEX" > "$DELDBFILE"

# Delete the index previously marked for deletion
echo "Deleting index $DEL_INDEX.."
curl -XDELETE "$DST/$DEL_INDEX"

echo "..Done."