Problem:
Explore the elasticsearch bulk API for doing batch operations.
Solution Summary:
We can perform operations in batches using the _bulk API.
Prerequisites:
Need to have a working elastic stack configuration with Elasticsearch and Kibana. You may also use any other HTTP clients instead of Kibana.
Solution Steps:
Case 1.a - Upload accounts.json to Elastic Cloud from Terminal (Mac)
We will upload a sample json dataset provided by elastic.co here.
-
Save the JSON file as accounts.json
-
Run following if in Mac: curl -u elastic:<password> -H "Content-Type: application/json" -X PUT <http-endpoint>/accounts/_doc/_bulk?pretty --data-binary "@accounts.json"
-
Non-mac users can either install cUrl or use any http client you are familiar with.
-
Account JSON Records will have following structure:
{"index":{"_id":"1"}}
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
Tip (Hack): If you use curl frequently to work with Elasticsearch, you can override the default curl within your home folder's bin directory to always add -H "Content-Type: application/json". Any script under home folder's bin directory takes precedence over other scripts.
Case 1.b - Add opening_date to 10 documents through bulk update from Kibana Devtools
POST accounts/_doc/_bulk
{ "update" : {"_id" : 1}}
{ "doc" : {"opening_date" : "2018/01/01"} }
{ "update" : {"_id" : 2}}
{ "doc" : {"opening_date" : "2018/02/04"} }
{ "update" : {"_id" : 3}}
{ "doc" : {"opening_date" : "2018/03/09"} }
{ "update" : {"_id" : 4}}
{ "doc" : {"opening_date" : "2018/04/16"} }
{ "update" : {"_id" : 5}}
{ "doc" : {"opening_date" : "2018/05/25"} }
{ "update" : {"_id" : 6}}
{ "doc" : {"opening_date" : "2018/01/01"} }
{ "update" : {"_id" : 7}}
{ "doc" : {"opening_date" : "2018/02/04"} }
{ "update" : {"_id" : 8}}
{ "doc" : {"opening_date" : "2018/03/09"} }
{ "update" : {"_id" : 9}}
{ "doc" : {"opening_date" : "2018/04/16"} }
{ "update" : {"_id" : 10}}
{ "doc" : {"opening_date" : "2018/05/25"} }
Note:
-
For update, line 2 needs a script or doc as given above. Index create/insert does not need (see next case).
Case 2 - Create / Insert with _bulk API from Console
POST /student/external/_bulk?pretty
POST /student/_doc/_bulk?pretty
{"index":{"_id":"1"}}
{"name": "John Doe", "age" : 20}
{"index":{"_id":"2"}}
{"name": "Doe John", "age" : 30}
{"index":{"_id":"3"}}
{"name": "Doe Doe", "age" : 40}
{"index":{"_id":"4"}}
{"name": "Doe Doe", "age" : 50}
Case 3 - Delete with _bulk API from Console
POST /student/_doc/_bulk?pretty
{"delete":{"_id":"2"}}
Recent comments