Our Work
200+ Enterprises across globally trust KTree for their Web & Mobile application Development needs.See What We Do
Updated 21 day ago
In this tutorial, we will mainly describe how to store AWS Elasticsearch snapshots in S3 buckets for later retrieval purposes. This process usually takes the following six steps
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::Your-S3-Bucket-Name"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"iam:PassRole"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::Your-S3-Bucket-Name/*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "es.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1",
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"arn:aws:iam::123456789012:role/RoleName"
]
}
]
}
Finally, we have all the required components ready, lets go ahead and Register snapshot repository.
from boto.connection import AWSAuthConnection
class ESConnection(AWSAuthConnection):
def __init__(self, region, **kwargs):
super(ESConnection, self).__init__(**kwargs)
self._set_auth_region_name(region)
self._set_auth_service_name("es")
def _required_auth_capability(self):
return ['hmac-v4']
if __name__ == "__main__":
client = ESConnection(
region='us-east-1',
host='your-elasticSearch-url',
aws_access_key_id='Give your access key',
aws_secret_access_key='Give your secret key', is_secure=False)
print 'Registering Snapshot Repository'
resp = client.make_request(method='POST',
path='/_snapshot/S3-Bucket-name',
data='{"type": "s3","settings": { "bucket": "es-prod-snapshot-backups","region": "us-east-1","role_arn": "give the arn of IAM role"}}')
body = resp.read()
print body
Note : Since this is an one time process, you can remove the python file and uninstall python boto package once this process is completed but you should not remove IAM user , Role or their policies.
To see how it works, run the python file for example,
#python registerSnapshot.py
You will see a message showing that repository registration successfully.
Next, you'll need to query the ElasticSearch from Sense console or CURL to take snapshot to S3 bucket and later you can restore these snapshots to ElasticSearch.
PUT /_snapshot/snapshot_repo_name/snapshot_name
{
"indices": "indice 1, indice 2, … indice n",
"ignore_unavailable": true,
"include_global_state": false
}
PUT /_snapshot/snapshot_repo_name/snapshot_name/_restore :
Note: If you are using curl, add ‘curl’ before the put request.