--print-tablet-detail
The --print-tablet-detail
switch takes no argument in order to lookup the tablet-detail from a live cluster,
or a snapshot number to lookup the tablet-detail from a snapshot.
Because the tablet detail information is fetched separately for each individual tablet, it is NOT fetched by default.
To let yb_stats
fetch the additional data for --print-tablet-detail
, you must add the --extra-data
switch!
The --print-tablet-detail
switch also requires the extra --uuid
switch to set the UUID for the tablet to print the details.
In order to obtain the UUID to use for this switch, use the --print-entities
option to obtain a list of tablets with their UUIDs.
Get a list of entities to obtain tablet UUID:
yb_stats --print-entities
Keyspace: ysql.postgres id: 000033e6000030008000000000000000
Keyspace: ysql.yugabyte id: 000033e8000030008000000000000000
Keyspace: ysql.system_platform id: 000033e9000030008000000000000000
Object: ysql.yugabyte.t, state: RUNNING, id: 000033e8000030008000000000004000
Tablet: ysql.yugabyte.t.4abf56bde0e843cfa9de8f48ca0e6a71 state: RUNNING
Replicas: (yb-1.local:9100(VOTER:LEADER), yb-3.local:9100(VOTER), yb-2.local:9100(VOTER),)
Tablet: ysql.yugabyte.t.56263119ab85438481a3b5865dfc5787 state: RUNNING
Replicas: (yb-1.local:9100(VOTER), yb-3.local:9100(VOTER), yb-2.local:9100(VOTER:LEADER),)
Tablet: ysql.yugabyte.t.acc6cf799c22457fb722ba9a361c54b9 state: RUNNING
Replicas: (yb-1.local:9100(VOTER), yb-3.local:9100(VOTER:LEADER), yb-2.local:9100(VOTER),)
For the table ysql.yugabyte.t, there are 3 tablets. Let's pick 4abf56bde0e843cfa9de8f48ca0e6a71.
yb_stats --print-tablet-detail --extra-data --uuid 4abf56bde0e843cfa9de8f48ca0e6a71
192.168.66.80:9000
General info:
Keyspace: yugabyte
Object name: t
On disk sizes: Total: 21.87M Consensus Metadata: 1.5K WAL Files: 2.00M SST Files: 19.87M SST Files Uncompressed: 189.64M
State: RUNNING
Consensus:
State: Consensus queue metrics:Only Majority Done Ops: 0, In Progress Ops: 0, Cache: LogCacheStats(num_ops=0, bytes=0, disk_reads=0)
Queue overview: Consensus queue metrics:Only Majority Done Ops: 0, In Progress Ops: 0, Cache: LogCacheStats(num_ops=0, bytes=0, disk_reads=0)
Watermark:
- { peer: 370474e547cc422ab838282184367b9b is_new: 0 last_received: 4.659 next_index: 660 last_known_committed_idx: 659 is_last_exchange_successful: 1 needs_remote_bootstrap: 0 member_type: VOTER num_sst_files: 4 last_applied: 4.659 }
- { peer: 3fc85b37b3fd4332bc7ed0bcf128b5de is_new: 0 last_received: 4.659 next_index: 660 last_known_committed_idx: 659 is_last_exchange_successful: 1 needs_remote_bootstrap: 0 member_type: VOTER num_sst_files: 2 last_applied: 4.659 }
- { peer: 35db27f008cb4a3ba7c7c5b224bacb7a is_new: 0 last_received: 4.659 next_index: 660 last_known_committed_idx: 659 is_last_exchange_successful: 1 needs_remote_bootstrap: 0 member_type: VOTER num_sst_files: 4 last_applied: 4.659 }
Messages:
- Entry: 0, Opid: 0.0, mesg. type: REPLICATE UNKNOWN_OP, size: 6, status: term: 0 index: 0
LogAnchor:
Latest log entry op id: 4.659
Min retryable request op id: 9223372036854775807.9223372036854775807
Last committed op id: 4.659
Earliest needed log index: 659
Transactions:
- { safe_time_for_participant: { physical: 1679241188204604 logical: 1 } remove_queue_size: 0 }
Rocksdb:
IntentDB:
RegularDB:
total_size: 2051458, uncompressed_size: 19655884, name_id: 14, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
total_size: 6273456, uncompressed_size: 59741998, name_id: 13, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
total_size: 6220244, uncompressed_size: 59741913, name_id: 12, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
total_size: 6291230, uncompressed_size: 59709058, name_id: 11, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
192.168.66.82:9000
General info:
Keyspace: yugabyte
Object name: t
On disk sizes: Total: 84.87M Consensus Metadata: 1.5K WAL Files: 65.00M SST Files: 19.87M SST Files Uncompressed: 189.64M
State: RUNNING
Consensus:
State: Consensus queue metrics:Only Majority Done Ops: 0, In Progress Ops: 1, Cache: LogCacheStats(num_ops=0, bytes=0, disk_reads=0)
Queue overview:
Watermark:
Messages:
LogAnchor:
Latest log entry op id: 4.659
Min retryable request op id: 9223372036854775807.9223372036854775807
Last committed op id: 4.659
Earliest needed log index: 659
Transactions:
- { safe_time_for_participant: { physical: 1679241188201243 } remove_queue_size: 0 }
Rocksdb:
IntentDB:
RegularDB:
total_size: 2051458, uncompressed_size: 19655884, name_id: 14, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
total_size: 6273456, uncompressed_size: 59741998, name_id: 13, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
total_size: 6220244, uncompressed_size: 59741913, name_id: 12, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
total_size: 6291230, uncompressed_size: 59709058, name_id: 11, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
192.168.66.81:9000
General info:
Keyspace: yugabyte
Object name: t
On disk sizes: Total: 84.87M Consensus Metadata: 1.5K WAL Files: 65.00M SST Files: 19.87M SST Files Uncompressed: 189.63M
State: RUNNING
Consensus:
State: Consensus queue metrics:Only Majority Done Ops: 0, In Progress Ops: 2, Cache: LogCacheStats(num_ops=0, bytes=0, disk_reads=1)
Queue overview:
Watermark:
Messages:
LogAnchor:
Latest log entry op id: 4.659
Min retryable request op id: 9223372036854775807.9223372036854775807
Last committed op id: 4.659
Earliest needed log index: 659
Transactions:
- { safe_time_for_participant: { physical: 1679241187538899 } remove_queue_size: 0 }
Rocksdb:
IntentDB:
RegularDB:
total_size: 5205489, uncompressed_size: 49526687, name_id: 12, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
total_size: 15627161, uncompressed_size: 149320345, name_id: 11, /mnt/d0/yb-data/tserver/data/rocksdb/table-000033e8000030008000000000004000/tablet-4abf56bde0e843cfa9de8f48ca0e6a71
This shows that a single tablet in the case of replication factor 3 is stored at multiple tablet servers. Therefore the first thing to show for a tablet (replica) is the hostname/ip address of the tablet server.
- General info:
- Keyspace: The YCQL keyspace or YSQL database name.
- Object name: the table or index name.
- On disk sizes: the sizes (with YugabyteDB accuracy) of this single tablet.
- State.
- Consensus:
- The state.
- Queue overview: this field only contains data if the replica is leader.
- Watermark: this will list the peers (including the local replica) for the tablet on the leader.
- Messages: this shows entries on the leader.
- Log Anchor:
- The log anchor data is internal logging state administration.
- Transactions:
- The transactions show HLC safe time, and might list other transactions on the leader when these are happening.
- Rocksdb:
- IntentDB: the file status of the SST files for the IntentDB.
- RegularDB: the file status of the SST files for the RegularDB. This is the actual data storage.
This is low level detail for troubleshooting.