Problem:
Explore the elasticsearch bulk API for doing batch operations with Amazon ES.
Solution Summary:
We can perform operations in batches using the _bulk API.
Prerequisites:
Need to have a working elastic stack configuration with Elasticsearch and Kibana. You may also use any other HTTP clients instead of Kibana.
Solution Steps:
Case 1.a - Upload accounts.json to Elastic Cloud from Terminal (Mac)
We will upload a sample json dataset provided by elastic.co here.
-
Save the JSON fileas accounts.json
-
Run following if in Mac: curl -H "Content-Type: application/json" -X PUT <http-endpoint>/accounts/_doc/_bulk?pretty --data-binary "@accounts.json"
-
Non-mac users car either install cUrl or use any http client you are familar with.
-
Note that we had not specified any authentication schemes except whitelisting our IP address.
-
Case 1.b - Add a field opening_date to 10 documents through bulk update from Kibana Devtools
POST accounts/_doc/_bulk
{ "update" : {"_id" : 1}}
{ "doc" : {"opening_date" : "2018/01/01"} }
{ "update" : {"_id" : 2}}
{ "doc" : {"opening_date" : "2018/02/04"} }
{ "update" : {"_id" : 3}}
{ "doc" : {"opening_date" : "2018/03/09"} }
{ "update" : {"_id" : 4}}
{ "doc" : {"opening_date" : "2018/04/16"} }
{ "update" : {"_id" : 5}}
{ "doc" : {"opening_date" : "2018/05/25"} }
{ "update" : {"_id" : 6}}
{ "doc" : {"opening_date" : "2018/01/01"} }
{ "update" : {"_id" : 7}}
{ "doc" : {"opening_date" : "2018/02/04"} }
{ "update" : {"_id" : 8}}
{ "doc" : {"opening_date" : "2018/03/09"} }
{ "update" : {"_id" : 9}}
{ "doc" : {"opening_date" : "2018/04/16"} }
{ "update" : {"_id" : 10}}
{ "doc" : {"opening_date" : "2018/05/25"} }
Note:
-
For update, line 2 needs a script or doc as given above. Index create/insert does not need (see next case).
Case 2 - Create / Insert with _bulk API from Console
POST /student/external/_bulk?pretty
POST /student/_doc/_bulk?pretty
{"index":{"_id":"1"}}
{"name": "John Doe", "age" : 20}
{"index":{"_id":"2"}}
{"name": "Doe John", "age" : 30}
{"index":{"_id":"3"}}
{"name": "Doe Doe", "age" : 40}
{"index":{"_id":"4"}}
{"name": "Doe Doe", "age" : 50}
Case 3 - Delete with _bulk API from Console
POST /student/_doc/_bulk?pretty
{"delete":{"_id":"2"}}
Additional Notes
-
In real world, you will also need to configure and use authentication while making requests against Amazon ES from command line. See bulk update note on Elastic Cloud for syntax to pass username and password.
-
Please practice recipes within notes from 11 at Elasticsearch Essentials.
Recent comments