In this tutorial, we will mainly describe how to store AWS Elasticsearch snapshots in S3 buckets for later retrieval purposes. This process usually takes the following six steps
Step 1
- Login to aws s3 console at https://aws.amazon.com/s3/
- Create a new s3 bucket if needed.
Step 2
- Goto IAM https://console.aws.amazon.com/iam/
- Goto Roles from the navigation and click on create new role
- After creating the role , Assign following policy and make a note of Role’s ARN.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::Your-S3-Bucket-Name"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"iam:PassRole"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::Your-S3-Bucket-Name/*"
]
}
]
}
Step 3
- Click on newly created Role and goto Trust relationships/
- Click on Edit trust relationship and replace existing policy with following policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "es.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Step 4
- Goto IAM and create an user by selecting access type as Programmatic access.
- Now click the user and Assign following policy by changing your Role ARN , which we created in step 2.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1",
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"arn:aws:iam::123456789012:role/RoleName"
]
}
]
}
Step 5.
Finally, we have all the required components ready, lets go ahead and Register snapshot repository.
- Login to any linux machine and install python boto package as below
- #apt-get install python-pip
- #git clone git://github.com/boto/boto.git
- #cd boto
- #python setup.py install
- Next, you’ll need to create a file called registerSnapshot.py and declare the following code.
from boto.connection import AWSAuthConnection
class ESConnection(AWSAuthConnection):
def __init__(self, region, **kwargs):
super(ESConnection, self).__init__(**kwargs)
self._set_auth_region_name(region)
self._set_auth_service_name("es")
def _required_auth_capability(self):
return [‘hmac-v4′]
if __name__ == "__main__":
client = ESConnection(
region=’us-east-1′,
host=’your-elasticSearch-url’,
aws_access_key_id=’Give your access key’,
aws_secret_access_key=’Give your secret key’, is_secure=False)
print ‘Registering Snapshot Repository’
resp = client.make_request(method=’POST’,
path=’/_snapshot/S3-Bucket-name’,
data='{"type": "s3","settings": { "bucket": "es-prod-snapshot-backups","region": "us-east-1","role_arn": "give the arn of IAM role"}}’)
body = resp.read()
print body
Note : Since this is an one time process, you can remove the python file and uninstall python boto package once this process is completed but you should not remove IAM user , Role or their policies.
To see how it works, run the python file for example,
#python registerSnapshot.py
You will see a message showing that repository registration successfully.
Step 6
Next, you’ll need to query the ElasticSearch from Sense console or CURL to take snapshot to S3 bucket and later you can restore these snapshots to ElasticSearch.
To take Snapshot from Sense Console:
PUT /_snapshot/snapshot_repo_name/snapshot_name
{
"indices": "indice 1, indice 2, … indice n",
"ignore_unavailable": true,
"include_global_state": false
}
To Restore Snapshot from Sense Console:
PUT /_snapshot/snapshot_repo_name/snapshot_name/_restore :
Note: If you are using curl, add ‘curl’ before the put request.