sstableloader — SSTable 일괄 로드

sstableloader — SSTable 일괄 로드

sstableloader는 지정한 디렉토리 <dir_path>에서 발견된 SSTable들을 설정된 클러스터로 일괄 로드(bulk-load)하는 도구예요. <dir_path>의 상위 디렉토리 이름이 대상 키스페이스/테이블 이름으로 사용됩니다. 예를 들어 ma-1-big-Data.db라는 SSTable을 keyspace1/standard1에 로드하려면, /path/to/keyspace1/standard1/ 디렉토리에 ma-1-big-Data.db와 ma-1-big-Index.db 파일을 함께 두어야 해요. 이 도구는 새로운 SSTable을 만들며, 원본으로 복사해 둔 파일은 정리하지 않습니다.

아래 나열된 옵션 중 일부는 기대한 대로 동작하지 않을 수 있는데, 그런 경우에는 특정 사용 사례에 맞는 우회 방법(workaround)을 함께 설명해 드릴게요.

로드할 SSTable 파일이 읽는 도중에 컴팩션되는 것을 피하려면, 데이터 디렉토리가 아닌 다른 키스페이스/테이블 경로에 파일을 두세요. 관련 문서: CASSANDRA-1278

이 도구 실행 전에 Cassandra를 반드시 중지해야 해요. 그렇지 않으면 예상치 못한 결과가 발생할 수 있으며, 스크립트는 Cassandra가 정말 중지되었는지 검증하지 않습니다.

출처: sstableloader — SSTable 일괄 로드

본문

기본 사용법은 다음과 같습니다.

sstableloader <options> <dir_path>

지원하는 옵션은 다음과 같습니다.

-f 명령줄 옵션으로 cassandra.yaml 파일을 제공하면 스트리밍 처리량과 클라이언트·서버 암호화 옵션을 설정할 수 있어요. yaml에서 읽히는 값은 stream_throughput_outbound, server_encryption_options, client_encryption_options뿐입니다. cassandra.yaml에서 읽은 옵션은 대응하는 명령줄 옵션으로 덮어쓸 수 있습니다.

스냅샷에서 SSTable 로드

스냅샷의 SSTable들을 접근 가능한 디렉토리로 복사한 뒤, sstableloader로 복원할 수 있어요.

예시:

cp snapshots/1535397029191/* /path/to/keyspace1/standard1/

sstableloader --nodes 172.17.0.2 /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/ma-3-big-Data.db to [/172.17.0.2]
progress: [/172.17.0.2]0:1/1 100% total: 100% 0  MB/s(avg: 1 MB/s)
Summary statistics:
   Connections per host:         : 1
   Total files transferred:      : 1
   Total bytes transferred:      : 4700000
   Total duration (ms):          : 4390
   Average transfer rate (MB/s): : 1
   Peak transfer rate (MB/s):    : 1

-d 또는 --nodes 옵션은 필수예요. 없으면 스크립트가 실행되지 않습니다.

예시:

sstableloader /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/
Initial hosts must be specified (-d)

SSL 클러스터를 위한 설정 파일 사용

클러스터에서 SSL 암호화가 켜져 있다면, --conf-path 옵션으로 관련 server_encryption_options(예: 트러스트스토어 위치, 알고리즘)가 담긴 cassandra.yaml을 sstableloader에 지정해 주세요. 위에 나열된 개별 ssl 옵션을 명령줄로 일일이 넘기는 것보다 이 방식이 더 잘 동작합니다.

예시:

sstableloader --nodes 172.17.0.2 --conf-path /etc/cassandra/cassandra.yaml /var/lib/cassandra/loadme/keyspace1/standard1-0974e5a0aa5811e8a0a06d2c86545d91/snapshots/
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /var/lib/cassandra/loadme/keyspace1/standard1-0974e5a0aa5811e8a0a06d2c86545d91/mc-1-big-Data.db  to [/172.17.0.2]
progress: [/172.17.0.2]0:0/1 1  % total: 1% 9.165KiB/s (avg: 9.165KiB/s)
progress: [/172.17.0.2]0:0/1 2  % total: 2% 5.147MiB/s (avg: 18.299KiB/s)
progress: [/172.17.0.2]0:0/1 4  % total: 4% 9.751MiB/s (avg: 27.423KiB/s)
progress: [/172.17.0.2]0:0/1 5  % total: 5% 8.203MiB/s (avg: 36.524KiB/s)
...
progress: [/172.17.0.2]0:1/1 100% total: 100% 0.000KiB/s (avg: 480.513KiB/s)

Summary statistics:
   Connections per host    : 1
   Total files transferred : 1
   Total bytes transferred : 4.387MiB
   Total duration          : 9356 ms
   Average transfer rate   : 480.105KiB/s
   Peak transfer rate      : 586.410KiB/s

진행 출력 숨기기

진행 상황과 요약 통계 출력을 숨기려면(예: 스크립트에서 이 도구를 사용할 때) --no-progress 옵션을 사용하세요.

예시:

sstableloader --nodes 172.17.0.2 --no-progress /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/ma-4-big-Data.db to [/172.17.0.2]

더 자세한 출력 얻기

--verbose 옵션을 사용하면 더 많은 진행 출력을 얻을 수 있어요.

예시:

sstableloader --nodes 172.17.0.2 --verbose /var/lib/cassandra/loadme/keyspace1/standard1-0974e5a0aa5811e8a0a06d2c86545d91/
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /var/lib/cassandra/loadme/keyspace1/standard1-0974e5a0aa5811e8a0a06d2c86545d91/mc-1-big-Data.db  to [/172.17.0.2]
progress: [/172.17.0.2]0:0/1 1  % total: 1% 12.056KiB/s (avg: 12.056KiB/s)
progress: [/172.17.0.2]0:0/1 2  % total: 2% 9.092MiB/s (avg: 24.081KiB/s)
progress: [/172.17.0.2]0:0/1 4  % total: 4% 18.832MiB/s (avg: 36.099KiB/s)
progress: [/172.17.0.2]0:0/1 5  % total: 5% 2.253MiB/s (avg: 47.882KiB/s)
progress: [/172.17.0.2]0:0/1 7  % total: 7% 6.388MiB/s (avg: 59.743KiB/s)
progress: [/172.17.0.2]0:0/1 8  % total: 8% 14.606MiB/s (avg: 71.635KiB/s)
progress: [/172.17.0.2]0:0/1 9  % total: 9% 8.880MiB/s (avg: 83.465KiB/s)
progress: [/172.17.0.2]0:0/1 11 % total: 11% 5.217MiB/s (avg: 95.176KiB/s)
progress: [/172.17.0.2]0:0/1 12 % total: 12% 12.563MiB/s (avg: 106.975KiB/s)
progress: [/172.17.0.2]0:0/1 14 % total: 14% 2.550MiB/s (avg: 118.322KiB/s)
progress: [/172.17.0.2]0:0/1 15 % total: 15% 16.638MiB/s (avg: 130.063KiB/s)
progress: [/172.17.0.2]0:0/1 17 % total: 17% 17.270MiB/s (avg: 141.793KiB/s)
progress: [/172.17.0.2]0:0/1 18 % total: 18% 11.280MiB/s (avg: 153.420KiB/s)
progress: [/172.17.0.2]0:0/1 19 % total: 19% 2.903MiB/s (avg: 164.603KiB/s)
progress: [/172.17.0.2]0:0/1 21 % total: 21% 6.744MiB/s (avg: 176.061KiB/s)
progress: [/172.17.0.2]0:0/1 22 % total: 22% 6.011MiB/s (avg: 187.440KiB/s)
progress: [/172.17.0.2]0:0/1 24 % total: 24% 9.690MiB/s (avg: 198.920KiB/s)
progress: [/172.17.0.2]0:0/1 25 % total: 25% 11.481MiB/s (avg: 210.412KiB/s)
progress: [/172.17.0.2]0:0/1 27 % total: 27% 9.957MiB/s (avg: 221.848KiB/s)
progress: [/172.17.0.2]0:0/1 28 % total: 28% 10.270MiB/s (avg: 233.265KiB/s)
progress: [/172.17.0.2]0:0/1 29 % total: 29% 7.812MiB/s (avg: 244.567KiB/s)
progress: [/172.17.0.2]0:0/1 31 % total: 31% 14.843MiB/s (avg: 256.021KiB/s)
progress: [/172.17.0.2]0:0/1 32 % total: 32% 11.457MiB/s (avg: 267.394KiB/s)
progress: [/172.17.0.2]0:0/1 34 % total: 34% 6.550MiB/s (avg: 278.536KiB/s)
progress: [/172.17.0.2]0:0/1 35 % total: 35% 9.115MiB/s (avg: 289.782KiB/s)
progress: [/172.17.0.2]0:0/1 37 % total: 37% 11.054MiB/s (avg: 301.064KiB/s)
progress: [/172.17.0.2]0:0/1 38 % total: 38% 10.449MiB/s (avg: 312.307KiB/s)
progress: [/172.17.0.2]0:0/1 39 % total: 39% 1.646MiB/s (avg: 321.665KiB/s)
progress: [/172.17.0.2]0:0/1 41 % total: 41% 13.300MiB/s (avg: 332.872KiB/s)
progress: [/172.17.0.2]0:0/1 42 % total: 42% 14.370MiB/s (avg: 344.082KiB/s)
progress: [/172.17.0.2]0:0/1 44 % total: 44% 16.734MiB/s (avg: 355.314KiB/s)
progress: [/172.17.0.2]0:0/1 45 % total: 45% 22.245MiB/s (avg: 366.592KiB/s)
progress: [/172.17.0.2]0:0/1 47 % total: 47% 25.561MiB/s (avg: 377.882KiB/s)
progress: [/172.17.0.2]0:0/1 48 % total: 48% 24.543MiB/s (avg: 389.155KiB/s)
progress: [/172.17.0.2]0:0/1 49 % total: 49% 4.894MiB/s (avg: 399.688KiB/s)
progress: [/172.17.0.2]0:0/1 51 % total: 51% 8.331MiB/s (avg: 410.559KiB/s)
progress: [/172.17.0.2]0:0/1 52 % total: 52% 5.771MiB/s (avg: 421.150KiB/s)
progress: [/172.17.0.2]0:0/1 54 % total: 54% 8.738MiB/s (avg: 431.983KiB/s)
progress: [/172.17.0.2]0:0/1 55 % total: 55% 3.406MiB/s (avg: 441.911KiB/s)
progress: [/172.17.0.2]0:0/1 56 % total: 56% 9.791MiB/s (avg: 452.730KiB/s)
progress: [/172.17.0.2]0:0/1 58 % total: 58% 3.401MiB/s (avg: 462.545KiB/s)
progress: [/172.17.0.2]0:0/1 59 % total: 59% 5.280MiB/s (avg: 472.840KiB/s)
progress: [/172.17.0.2]0:0/1 61 % total: 61% 12.232MiB/s (avg: 483.663KiB/s)
progress: [/172.17.0.2]0:0/1 62 % total: 62% 9.258MiB/s (avg: 494.325KiB/s)
progress: [/172.17.0.2]0:0/1 64 % total: 64% 2.877MiB/s (avg: 503.640KiB/s)
progress: [/172.17.0.2]0:0/1 65 % total: 65% 7.461MiB/s (avg: 514.078KiB/s)
progress: [/172.17.0.2]0:0/1 66 % total: 66% 24.247MiB/s (avg: 525.018KiB/s)
progress: [/172.17.0.2]0:0/1 68 % total: 68% 9.348MiB/s (avg: 535.563KiB/s)
progress: [/172.17.0.2]0:0/1 69 % total: 69% 5.130MiB/s (avg: 545.563KiB/s)
progress: [/172.17.0.2]0:0/1 71 % total: 71% 19.861MiB/s (avg: 556.392KiB/s)
progress: [/172.17.0.2]0:0/1 72 % total: 72% 15.501MiB/s (avg: 567.122KiB/s)
progress: [/172.17.0.2]0:0/1 74 % total: 74% 5.031MiB/s (avg: 576.996KiB/s)
progress: [/172.17.0.2]0:0/1 75 % total: 75% 22.771MiB/s (avg: 587.813KiB/s)
progress: [/172.17.0.2]0:0/1 76 % total: 76% 22.780MiB/s (avg: 598.619KiB/s)
progress: [/172.17.0.2]0:0/1 78 % total: 78% 20.684MiB/s (avg: 609.386KiB/s)
progress: [/172.17.0.2]0:0/1 79 % total: 79% 22.920MiB/s (avg: 620.173KiB/s)
progress: [/172.17.0.2]0:0/1 81 % total: 81% 7.458MiB/s (avg: 630.333KiB/s)
progress: [/172.17.0.2]0:0/1 82 % total: 82% 22.993MiB/s (avg: 641.090KiB/s)
progress: [/172.17.0.2]0:0/1 84 % total: 84% 21.392MiB/s (avg: 651.814KiB/s)
progress: [/172.17.0.2]0:0/1 85 % total: 85% 7.732MiB/s (avg: 661.938KiB/s)
progress: [/172.17.0.2]0:0/1 86 % total: 86% 3.476MiB/s (avg: 670.892KiB/s)
progress: [/172.17.0.2]0:0/1 88 % total: 88% 19.889MiB/s (avg: 681.521KiB/s)
progress: [/172.17.0.2]0:0/1 89 % total: 89% 21.077MiB/s (avg: 692.162KiB/s)
progress: [/172.17.0.2]0:0/1 91 % total: 91% 24.062MiB/s (avg: 702.835KiB/s)
progress: [/172.17.0.2]0:0/1 92 % total: 92% 19.798MiB/s (avg: 713.431KiB/s)
progress: [/172.17.0.2]0:0/1 94 % total: 94% 17.591MiB/s (avg: 723.965KiB/s)
progress: [/172.17.0.2]0:0/1 95 % total: 95% 13.725MiB/s (avg: 734.361KiB/s)
progress: [/172.17.0.2]0:0/1 96 % total: 96% 16.737MiB/s (avg: 744.846KiB/s)
progress: [/172.17.0.2]0:0/1 98 % total: 98% 22.701MiB/s (avg: 755.443KiB/s)
progress: [/172.17.0.2]0:0/1 99 % total: 99% 18.718MiB/s (avg: 765.954KiB/s)
progress: [/172.17.0.2]0:1/1 100% total: 100% 6.613MiB/s (avg: 767.802KiB/s)
progress: [/172.17.0.2]0:1/1 100% total: 100% 0.000KiB/s (avg: 670.295KiB/s)

Summary statistics:
   Connections per host    : 1
   Total files transferred : 1
   Total bytes transferred : 4.387MiB
   Total duration          : 6706 ms
   Average transfer rate   : 669.835KiB/s
   Peak transfer rate      : 767.802KiB/s

로드 스로틀링

테이블 로더가 시스템 리소스를 과도하게 사용하지 않도록 --throttle 옵션으로 프로세스 속도를 제한할 수 있어요. 기본값은 무제한(스로틀 없음)이며, 단위는 메가비트(megabits)입니다. 아래 예시에서 총 소요 시간이 늘어난 것을 확인할 수 있어요.

예시:

sstableloader --nodes 172.17.0.2 --throttle 1 /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/ma-6-big-Data.db to [/172.17.0.2]
progress: [/172.17.0.2]0:1/1 100% total: 100% 0  MB/s(avg: 0 MB/s)
Summary statistics:
   Connections per host:         : 1
   Total files transferred:      : 1
   Total bytes transferred:      : 4595705
   Total duration (ms):          : 37634
   Average transfer rate (MB/s): : 0
   Peak transfer rate (MB/s):    : 0

로드 속도 높이기

로드 과정을 빠르게 하려면 호스트당 연결 수를 늘릴 수 있어요.

예시:

sstableloader --nodes 172.17.0.2 --connections-per-host 100 /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of /var/lib/cassandra/loadme/keyspace1/standard1-f8a4fa30aa2a11e8af27091830ac5256/ma-9-big-Data.db to [/172.17.0.2]
progress: [/172.17.0.2]0:1/1 100% total: 100% 0  MB/s(avg: 1 MB/s)
Summary statistics:
   Connections per host:         : 100
   Total files transferred:      : 1
   Total bytes transferred:      : 4595705
   Total duration (ms):          : 3486
   Average transfer rate (MB/s): : 1
   Peak transfer rate (MB/s):    : 1

이렇게 작은 데이터셋은 호스트당 연결 수를 늘려도 큰 효과가 없지만, 예시에서 총 소요 시간이 줄어든 것을 알 수 있어요.

더 알아보기 (Learn more)

• SSTable 도구 개요
• nodetool snapshot — 스냅샷 생성