Skip to content

Releases: medcl/esm

v0.7.0

28 Feb 08:51
628122f
Compare
Choose a tag to compare

Changes

  • Add sync function for incremental update #81 @fishjam
  • Add go.mod and fix build error(infini.sh/framework, fasthttp, util) #82 @fishjam

v0.6.1

04 Mar 01:34
99b7728
Compare
Choose a tag to compare

Decrease default settings to use less memory first.

v0.6.0

17 Feb 08:16
Compare
Choose a tag to compare

Fix 413 Request Entity Too Large

v0.5.0

19 Dec 08:57
d6402a7
Compare
Choose a tag to compare
  • Add buffer_count to control memory consumed by ESM
  • Speedup, several performance optimization

A 3 nodes cluster(3 * c5d.4xlarge, 16C,32GB,10Gbps)

./esm -s https://localhost:8000 -d https://localhost:8000 -x kibana_sample_data_logs -y logs-test -m elastic:medcl123 -n elastic:medcl123  --regenerate_id --repeat_times=5
./esm -s https://localhost:8000 -d https://localhost:8000 -x logs-test -y logs-test1 -m elastic:medcl123 -n elastic:medcl123  --regenerate_id  -w 20 --sliced_scroll_size=20 -b 20 --repeat_times=500
./esm -s https://localhost:8000 -d https://localhost:8000 -x logs-test1 -y logs-test -m elastic:medcl123 -n elastic:medcl123  --regenerate_id  -w 20 --sliced_scroll_size=20 -b 20 --repeat_times=500

[12-19 06:29:40] [INF] [main.go:537,main] data migration finished.
root@ip-172-31-13-181:/tmp# ./esm -s https://localhost:8000 -d https://localhost:8000 -x logs1kw -y logs122 -m elastic:medcl123 -n elastic:medcl123 -w 40 --sliced_scroll_size=60 -b 5 --buffer_count=2000000  --regenerate_id
[12-19 06:31:20] [INF] [main.go:506,main] start data migration..
Scroll 10064570 / 10064570 [============================================] 100.00% 55s
Bulk 10062602 / 10064570 [=============================================]  99.98% 55s
[12-19 06:32:15] [INF] [main.go:537,main] data migration finished.

Migrate 10,000,000 documents within 1 minute (Nginx log generated from kibana_sample_data_logs).

v0.4.5

14 May 13:00
d6402a7
Compare
Choose a tag to compare
  • Fix big numbers in JSON without loss of precision, fix #36
  • Fix total structure change for Elasticsearch 6.x

v0.4.4

14 May 07:04
Compare
Choose a tag to compare
  • Support run in background, fix #37
  • Support generate data from source
  • Support fields rename
  • Fix routing issue for version > 6.x
  • Fix source fields select issue on 2.x

0.4.4_SNAPSHOT

17 Jan 11:09
Compare
Choose a tag to compare
0.4.4_SNAPSHOT Pre-release
Pre-release

0.4.4_SNAPSHOT

v0.4.3

17 Sep 08:28
Compare
Choose a tag to compare
  • enable to skip cluster status check, use index instead of create for bulk operations
  • support migrate 7.x to 2.x
  • expose bulk error, support to rename field
  • fix issue for es 7.x
  • ignore insecure ssl, support override type name, fix connection leak

v0.4.2

22 Apr 04:54
Compare
Choose a tag to compare
update example

v0.4.1

02 Aug 03:59
Compare
Choose a tag to compare
dirty fix for es v6, Closes #15