¶ STORAGE PERFORMANCE TESTING FRAMEWORK - REV 1.1.0
¶ TABLE OF CONTENTS
- INTRODUCTION TO THE FRAMEWORK
- TESTING METHODOLOGY
- FRAMEWORK TOOLS
- STRATISYSTEM NODE SPECIFICATIONS
- NETWORK SMOKE TEST
- FRAMEWORK TOOL DETAILS
- INSTALLING THE TOOLS
- PROTOCOLS
- MOUNTING VARIOUS PROTOCOL TYPES
- NFS
- SMB
- CephFS
- Ceph RBD
- iSCSI
- NVMeoF
- RUNNING THE TESTS
- PERFPROFILER
- FRAMETEST
- IOR
- FIO
- S3 WARP
- PERFORMANCE RESULTS
- TEST SYSTEM HARDWARE
- IOR TEST RESULTS (4-NODE HCI)
- IOR TEST RESULTS (8-NODE HCI)
- FRAMETEST RESULTS (8-NODE HCI)
- FIO TEST RESULTS (SINGLE VM INSTANCE - DIRECT I/O CACHE BYPASS)
- CONCLUSION
¶ INTRODUCTION TO THE FRAMEWORK
This document presents a detailed testing framework for evaluating storage system performance using five specialized tools: Perfprofiler, IOR, FIO, frametest, and S3 Warp. Each tool addresses different facets of storage performance, from basic I/O operations to parallel file systems, media frame handling, and object storage benchmarking. Utilizing these tools, we aim to provide a holistic assessment that aligns with real-world usage scenarios, ensuring that the results are both reliable and actionable. This framework is particularly relevant for IT professionals, system administrators, and engineers working on storage optimization, especially in high-performance computing (HPC), media production, and cloud storage environments.
We will explore each tool’s purpose, functionality, and application, followed by a structured testing methodology that incorporates best practices for producing meaningful test results. Additionally, we expand the framework to include guidance on hyper-converged testing and tiered over-the-wire testing, incorporating protocols such as NFS, SMB, iSCSI, NVMe-oF, and S3, to address modern storage architectures and network-based access.
¶ TESTING METHODOLOGY
The methodology outlined in this document includes:
- Manifest of tools used in this framework
- Minimum recommended hardware requirements
- Installing the test tools (where necessary)
- Running the performance tests with best practice tool parameters
- Analyzing the results
¶ FRAMEWORK TOOLS
- PERFPROFILER: A comprehensive FIO-based tool for generating full range performance metrics for read, write, random, sequential across all block sizes. Generates data which can be used to generate a 3D performance graph.
- IOR: A parallel I/O benchmark for HPC file systems, using MPI for distributed testing.
- FIO: A flexible I/O tester for simulating storage workloads, providing metrics like IOPS and bandwidth.
- FRAMETEST: Focuses on media frame performance, simulating video production workloads.
- S3 WARP: Benchmarks S3-compatible object storage, measuring throughput and latency.
STRATISYSTEM NODE SPECIFICATIONS
| COMPONENT | OPTIONS |
|---|---|
| CPU |
|
| MEMORY |
|
| NETWORK |
|
| STORAGE |
|
¶ NETWORK SMOKE TEST
Performing an initial network smoke test (aka. basic capabilities test) is an important step to ensuring that the networking components of the infrastructure are configured and functioning properly. For this test, it is recommended that iperf3 is used in the following manner:
On the server node:
for port in {5101..5108}; do iperf3 -s -p $port & doneNOTE: This command will activate iperf3 on the server node, listening on 8 channels. If higher bandwidth cards are used (400Gb or higher), you will likely need as many as 24 channels to saturate the links.
On the client node:
for i in {1..8}; do iperf3 -c 192.168.100.43 -w1M -Z –s$i -p $((5100 + i)) & done; waitNOTE: This command will activate iperf3 on the client node, driving traffic over 8 channels to the server node. The number of channels should match the number of channels configured on the server node
The expectation of the iperf3 test is that at least 90% of line speed is achieved. If this is not achieved, investigation into the network should be conducted (e.g. drivers, settings, hardware, etc.).
¶ FRAMEWORK TOOL DETAILS
¶ PERFPROFILER
SteelDome’s Perfprofiler tool uses FIO as the backend load generator. It is able to perform all test scenarios and output the raw data. Additionally, it generates data which can be used to generate 3D performance profile graphs which is useful for displaying the full spectrum of test combinations in a convenient easy to understand format.
How It Works: Perfprofiler spawns threads or processes based on user-specified actions, defined by runtime input parameters. It supports both synchronous and asynchronous operations, providing metrics like IOPS, bandwidth, and latency.
Usage in Testing Framework: Perfprofiler is ideal for benchmarking block storage and file systems, testing aspects like sequential and random reads/writes. It can be configured to mimic application-specific workloads, ensuring the test results reflect actual usage. Its versatility extends to network-mounted filesystems, making it suitable for over-the-wire testing with protocols like NFS and SMB.
For more details, refer to the official SteelDome documentation.
¶ FIO: FLEXIBLE I/O TESTER
FIO, or Flexible I/O Tester, is a versatile tool designed to simulate a wide range of I/O workloads for storage performance testing. It was originally developed to eliminate the need for custom test programs when evaluating specific workloads for performance or bug reproduction. FIO can handle various scenarios, including multiple processes or threads, memory-mapped files, and asynchronous I/O, making it suitable for testing disk and storage system performance.
How It Works: FIO spawns threads or processes based on user-specified actions, defined in a job file. This file outlines parameters such as read/write mix, block size, queue depth, and runtime, allowing for flexible simulation of real-world I/O patterns. It supports both synchronous and asynchronous operations, providing metrics like IOPS, bandwidth, and latency.
Usage in Testing Framework: FIO is ideal for benchmarking block storage and file systems, testing aspects like sequential and random reads/writes. It can be configured to mimic application-specific workloads, ensuring the test results reflect actual usage. Its versatility extends to network-mounted filesystems, making it suitable for over-the-wire testing with protocols like NFS and SMB.
For more details, refer to the official documentation at the FIO GitHub repo.
¶ IOR: INTERLEAVED OR RANDOM BENCHMARK
IOR (Interleaved or Random) is a parallel I/O benchmark tool tailored for testing the performance of parallel file systems, particularly in HPC environments. It uses the Message Passing Interface (MPI) for process synchronization, making it suitable for distributed systems.
How It Works: IOR runs multiple processes across client nodes, simulating I/O operations with configurable parameters such as file size, block size, and access patterns (sequential or random). It provides performance metrics like aggregate throughput and per-process statistics, essential for evaluating parallel file system efficiency.
Usage in Testing Framework: IOR is crucial for testing distributed storage systems, especially in HPC clusters, to assess scalability and performance under parallel workloads. It helps compare different file systems and identify bottlenecks in distributed environments, making it suitable for hyper-converged testing in multi-node setups.
For more details, refer to the official documentation at the IOR GitHub repo.
¶ FRAMETEST: MULTIMEDIA PERFORMANCE TOOL
Frametest is a specialized tool for testing storage performance in the context of handling media frames, such as those used in video production. It simulates writing and reading media frames to and from a disk, making it ideal for assessing storage systems in media-intensive applications.
How It Works: Frametest generates streaming workloads for various video formats (e.g., SD, HD, 4K), storing each frame in a separate file. It measures performance metrics like frame rates and throughput, simulating real-world media production scenarios. Parameters include frame size, number of frames, and test duration, with outputs indicating performance for reading and writing operations.
Usage in Testing Framework: frametest is used to evaluate storage performance for media workflows, ensuring the system can handle high frame rates and large file sizes typical in video editing and streaming. It’s particularly useful for identifying storage limitations in multimedia environments, and can be applied to network-mounted filesystems for over-the-wire testing.
Details can be found at the GitHub repository Frametest GitHub Tool for Media Frame Testing.
¶ S3 WARP: S3 BENCHMARK TOOL
S3 warp, developed by Minio, is a benchmarking tool for testing the performance of S3-compatible object storage systems. It is designed to measure the efficiency of operations like PUT and GET, assessing throughput, latency, and scalability in cloud storage environments. It supports configuration via command-line parameters or environment variables, making it flexible for various testing scenarios.
How It Works: S3 warp performs operations on a specified S3 endpoint, using credentials to create, delete, and list buckets, and upload files. It can be configured for different concurrency levels, object sizes, and operations, with results output in formats like JSON or compressed CSV files. By default, it uses a bucket named “warp-benchmark-bucket,” which is cleaned before and after each run.
Usage in Testing Framework: S3 warp is essential for testing object storage performance, particularly in cloud-based or S3-compatible systems. It helps evaluate scalability and performance under varying loads, crucial for data lake and AI/ML workloads, and is directly applicable for over-the-wire testing using the S3 protocol.
For more information, visit the GitHub page at S3 warp GitHub Minio Benchmarking Tool.
¶ INSTALLING THE TOOLS
¶ PERFPROFILER
This tool is already present on the StratiSYSTEM nodes. It is located in the base directory (/stratistor or /stratiserv) under the modules -> perftools -> perfprofiler directory. It can be made globally accessible by issuing the following command:
ln -s /stratistor/modules/perftools/perfprofiler/perfprofiler /usr/local/bin/perfprofilerNOTE: If the tool is to be used with other systems not using the StratiSYSTEM operating system, make sure that the FIO tools is also installed as it is a dependency of Perfprofiler.
¶ IOR
This tool is already present on the StratiSYSTEM nodes. It is located in the base directory (/stratistor or /stratiserv) under the modules -> perftools -> ior directory.
Perform on all nodes running the IOR tool:
dnf install -y openmpi-devel
ln -s /usr/lib64/openmpi/bin/mpirun /usr/local/bin/mpirun
ln -s /usr/lib64/openmpi/bin/orted /usr/local/bin/ortedNOTE: If the tool is to be used with other systems not using the StratiSYSTEM operating system, make sure that the IOR binaries (ior, mdtest, and md-workbench) are present on the test system.
¶ FIO
This tool is already installed on the StratiSYSTEM nodes. It is located in the /usr/bin directory.
NOTE: If the tool is to be used with other systems not using the StratiSYSTEM operating system, the tool can be installed by issuing the following command:
dnf install -y fio¶
FRAMETEST
This tool is already present on the StratiSYSTEM nodes. It is located in the base directory (/stratistor or /stratiserv) under the modules -> perftools -> frametest directory. It can be made globally accessible by issuing the following command:
ln -s /stratistor/modules/perftools/frametest/frametest /usr/local/bin/frametestNOTE: If the tool is to be used with other systems not using the StratiSYSTEM operating system, make sure that the frametest binary is located somewhere on the test system.
¶ S3 WARP
This tool is already present on the StratiSYSTEM nodes, but needs to be installed from RPM. The RPM is located in the base directory (/stratistor or /stratiserv) under the modules -> perftools -> s3warp directory. To install the tool, issue the following command:
rpm -ivh s3_warp_linux_x86_64.rpmNOTE: If the tool is to be used with other systems not using the StratiSYSTEM operating system, the RPM can be installed on the test system using the same command as shown above.
¶ PROTOCOLS
¶ NFS AND SMB
File-level storage protocols that enable network-based file sharing across multiple clients. They are widely used for collaborative access, allowing multiple users and applications to interact with shared data over a network. NFS is commonly used in Linux and Unix environments, while SMB is the standard for Windows-based file sharing. Performance considerations include network stability, concurrent access patterns, and filesystem-level caching optimizations.
¶ CEPHFS
A distributed file system that provides scalable, POSIX-compliant shared storage over a Ceph cluster. It is designed for high availability and resilience, automatically balancing data across multiple storage nodes. CephFS is well-suited for large-scale deployments requiring fault tolerance and supports metadata-intensive workloads, making it ideal for cloud platforms, HPC environments, and large-scale file repositories.
¶ CEPH RBD
A block storage solution built on Ceph, designed for virtual machines, containerized environments, and high-performance applications. It provides thin provisioning, snapshotting, and replication capabilities, making it a flexible alternative to traditional SAN storage. Ceph RBD scales dynamically across a cluster, distributing data for redundancy and resilience, ensuring reliability for cloud and enterprise workloads.
¶ ISCSI
A block-level storage protocol that enables remote access to storage devices over a network. It is widely used in enterprise environments for centralized storage solutions, such as databases, virtualization, and backup systems. iSCSI supports high queue depths and multipath I/O, making it suitable for handling heavy workloads while ensuring consistent throughput and low latency.
¶ NVMeoF
A next-generation block-level storage protocol designed for high-performance computing, database workloads, and low-latency applications. It extends NVMe capabilities over network fabrics, such as RDMA, TCP, and Fibre Channel, enabling faster access to remote storage devices with minimal overhead. NVMe-oF significantly improves IOPS and reduces latency, making it ideal for demanding workloads like AI, big data analytics, and real-time processing.
¶ S3
A scalable object storage protocol designed for cloud and distributed storage environments. It supports efficient storage and retrieval of unstructured data, such as backups, logs, media files, and large datasets. S3 is optimized for high concurrency and global accessibility, making it well-suited for cloud-native applications, big data storage, and content delivery networks.
Table: Tool Comparison
| TOOL NAME | FOCUS AREA | KEY METRICS | USE CASES | APPLICABLE PROTOCOLS |
|---|---|---|---|---|
| PERFPROFILER | General I/O Workloads | IOps, throughput, latency | Disk and file system benchmarking | NFS, SMB, iSCSI, NVMeoF, CephFS (parallel access) |
| IOR | Parallel File Systems | Throughput, Per-process Stats | HPC cluster storage testing | NFS, SMB, CephFS (parallel access) |
| FIO | General I/O Workloads | IOps, throughput, latency | Disk and file system benchmarking | NFS, SMB, iSCSI, NVMeoF, CephFS (parallel access) |
| FRAMETEST | Media Frames | Framerate, throughput, latency | Video production storage testing | NFS, SMB, CephFS (parallel access) |
| S3 WARP | Object Storage | PUT/GET performance, scalability | Cloud and S3-compatible storage | S3 Object |
¶ MOUNTING VARIOUS PROTOCOL TYPES
¶ NFS SETUP - MOUNT NFS EXPORT (NFSv4.1)
TEST COMMAND:
mount -t nfs4 -o rw,noatime,nodiratime,async,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys, \
lookupcache=none,noac,nolock,local_lock=none [STRATISTOR_IP_ADDR]:/nfs01 /mnt/[LOCAL MOUNT DIR]
¶ SMB SETUP - MOUNT SMB SHARE (SMBv3.11)
TEST COMMAND:
mount -t cifs -o username=your_username,password=your_password //server_address/share_name /mnt/smbshare
¶ CEPHFS SETUP - MOUNTS A CEPHFS FILESYSTEM
TEST COMMAND:
mount -t ceph {monitor-ip}:6789:/ /mnt/cephfs -o name={ceph-user},secret={ceph-key}
¶ CEPH RBD SETUP - MOUNTS A CEPH RBD BLOCK DEVICE
Commands:
rbd map rbddisk01 --pool iscsi01.data -k /etc/ceph/ceph.client.admin.keyring
mkdir /mnt/rbddisk01
mkfs.xfs -f /dev/rbd0
mount -t xfs /dev/rbd0 /mnt/rbddisk01
¶ ISCSI SETUP - MOUNTS AN ISCSI BLOCK DEVICE
Commands:
- DISCOVERY:
iscsiadm -m discovery -t sendtargets -p <TARGET_IP>
- LOGIN:
iscsiadm -m node -T iqn.2023-03.com.example:target1 -p <TARGET_IP>:3260 --login
¶ NVMEOF SETUP - MOUNTS A NVME BLOCK DEVICE
Commands:
- DISCOVERY:
nvme discover -t tcp -a <TARGET_IP> -s 4420
- LOGIN:
nvme connect -t tcp -n nqn.2023-03.com.example:nvme-target1 -a <TARGET_IP> -s 4420¶ RUNNING THE TESTS
¶ PERFPROFILER
TEST COMMAND:
perfprofiler [blocksizes] [patterns] [iodepth] [runtime] [threads] [startdelay] [directio] [numfiles] [savecsv] [size] [testlocation] [statfilelocation] <blocksizes> Comma-separated list of block sizes (e.g., 4K,16K) or 'all' for all block sizes (4K-4M)
<patterns> Test patterns: 'random', 'sequential', or 'all' for all patterns
<iodepth> IO depth [default: 32]
<runtime> Runtime for each test in seconds [default: 60]
<threads> Number of threads [default: 4]
<startdelay> Start delay in seconds [default: 0]
<directio> Bypass cache (0 - no, 1 - yes) [default: 1]
<numfiles> Number of test files [default: 1]
<savecsv> Save output in CSV format (0 - no, 1 - yes) [default: 1]
<size> Size of the working set (e.g., 16G, 32G, 128G)
<testlocation> Test device or file (e.g., /dev/sdz or /tmp/testfile)
<statfilelocation> Location to save stats file (e.g., /tmp/stats_file.json)
EXAMPLE TEST RUN
./perfprofiler all all 64 10 8 2 1 8 1 2G /tmp/testfile /tmp/stats_file.json
SteelDome Storage Profiler Tool v1.3
Test executing with the following parameters:
BLOCK SIZE: all
PATTERN: all
TEST SIZE: 2G
IO DEPTH: 64
PATTERN RUNTIME: 10 sec(s)
THREADS: 8 threads
START DELAY: 2 sec(s)
DIRECT IO: YES (Bypass cache)
TEST FILES: 8 file(s)
TEST LOCATION: /tmp/[auto-generated]
STATS LOCATION: /tmp/stats_file.json
CSV OUTPUT: YES
Pattern Operation Block Size IOPS Thruput (MB/s) Latency (ms)
--------------------------------------------------------------------------------------
write-random write 4K 82012 320 6.197
write-random write 8K 76021 594 6.684
write-random write 16K 91273 1426 5.597
write-random write 32K 74856 2339 6.826
write-random write 64K 37675 2355 13.567
write-random write 128K 17072 2134 29.944
write-random write 256K 8140 2035 62.748
write-random write 512K 3523 1761 145.095
write-random write 1M 2380 2380 214.049
write-random write 2M 1241 2482 411.163
write-random write 4M 625 2500 808.705
read-random read 4K 188152 735 2.718
read-random read 8K 167145 1306 3.060
read-random read 16K 148142 2315 3.452
read-random read 32K 138574 4330 3.691
read-random read 64K 86128 5383 5.940
read-random read 128K 41811 5226 12.233
read-random read 256K 20993 5248 24.366
read-random read 512K 6458 3229 79.073
read-random read 1M 5465 5465 93.363
read-random read 2M 5326 10652 95.801
read-random read 4M 3247 12988 156.566
write-sequential write 4K 378626 1479 1.349
write-sequential write 8K 231467 1808 2.207
write-sequential write 16K 126857 1982 4.027
write-sequential write 32K 68329 2135 7.480
write-sequential write 64K 35010 2188 14.600
write-sequential write 128K 16376 2047 31.218
write-sequential write 256K 8188 2047 62.413
write-sequential write 512K 4265 2132 119.766
write-sequential write 1M 2163 2163 235.850
write-sequential write 2M 1035 2070 491.282
write-sequential write 4M 544 2174 912.029
read-sequential read 4K 443858 1734 1.152
read-sequential read 8K 371094 2899 1.377
read-sequential read 16K 287801 4497 1.776
read-sequential read 32K 250766 7836 2.039
read-sequential read 64K 183566 11473 2.786
read-sequential read 128K 114879 14360 4.452
read-sequential read 256K 71103 17776 7.193
read-sequential read 512K 40824 20412 12.523
read-sequential read 1M 21028 21028 24.298
read-sequential read 2M 8801 17603 57.883
read-sequential read 4M 3431 13723 146.434
The CSV output can be used to easily generate a 3D performance profile providing a graphical snapshot of the full spectrum of performance across all performance parameters.
IO PERFORMANCE PROFILE
THROUGHPUT PERFORMANCE PROFILE
¶ FRAMETEST
WRITE TEST
TEST COMMAND:
frametest -w 4k -n10000 -t12 .Explanation:
- Performs write frame test operations.
- Uses a 4K framesize (~ 52MB each), 10,000 iterations, and 12 threads.
READ TEST
TEST COMMAND:
frametest -r 4k -n10000 -t12 .Explanation:
- Performs read frame test operations.
- Uses a 4K framesize (~ 52MB each), 10,000 iterations, and 12 threads.
NOTE: You must first run frametest write operations before running read operations as the read operation requires the frametest files to be in place.
EXAMPLE TEST RUN:
frametest -r 4k -n 5000 -t 20 .
Test parameters: -r -z49856 -n5000 -t20
Test duration: 18 secs
Frames transferred: 4918 (239445.125 MB)
Fastest frame: 52.669 ms (924.41 MB/s)
Slowest frame: 132.407 ms (367.71 MB/s)
Averaged details:
Open I/O Frame Data rate Frame rate
Last 1s: 0.454 ms 68.47 ms 3.64 ms 13369.12 MB/s 274.6 fps
5s: 0.436 ms 67.42 ms 3.69 ms 13191.83 MB/s 270.9 fps
30s: 0.382 ms 67.74 ms 3.67 ms 13276.03 MB/s 272.7 fps
Overall: 0.382 ms 67.74 ms 3.67 ms 13276.03 MB/s 272.7 fps
Histogram of frame completion times:
100% |
|
|
|
| *
| *
| *
| **
| **
| *******
+|----|-----|----|----|-----|----|----|-----|----|----|-----|----|
ms <0.1 .2 .5 1 2 5 10 20 50 100 200 500 >1s
Overall frame rate ..... 273.02 fps (13938416546 bytes/s)
Average file time ...... 68.095 ms
Shortest file time ..... 52.669 ms
Longest file time ...... 132.407 ms
Average open time ...... 0.380 ms
Shortest open time ..... 0.017 ms
Longest open time ...... 8.824 ms
Average read time ...... 67.7 ms
Shortest read time ..... 52.5 ms
Longest read time ...... 132.2 ms
Average close time ..... 0.006 ms
Shortest close time .... 0.001 ms
Longest close time ..... 0.027 ms
¶ IOR
1. Small Sequential I/O Size Test (4 KiB)
TEST COMMAND:
mpirun -mca btl tcp -np 128 -H hyper01:16,hyper02:16,hyper03:16,hyper04:16,hyper05:16,hyper06:16,hyper07:16,hyper08:16 \
/stratistor/modules/perftools/ior/ior -t 4k -b 256m -s 64 -F -C -e -i 2 -o /stratistor/clustermounts/machines/ior-test/testfileExplanation:
Executes the IOR tool across eight (8) test nodes with a total of 128 tasks, 16 clients per node, using 64 segments, with a transfer size of 4 KiB, block size of 256 MiB, and a total aggregate working set size of 2 TiB.
2. Medium Sequential I/O Size Test (32 KiB)
TEST COMMAND:
mpirun -mca btl tcp -np 128 -H hyper01:16,hyper02:16,hyper03:16,hyper04:16,hyper05:16,hyper06:16,hyper07:16,hyper08:16 \
/stratistor/modules/perftools/ior/ior -t 32k -b 256m -s 64 -F -C -e -i 2 -o /stratistor/clustermounts/machines/ior-test/testfileExplanation:
Executes the IOR tool across eight (8) test nodes with a total of 128 tasks, 16 clients per node, using 64 segments, with a transfer size of 32 KiB, block size of 256 MiB, and a total aggregate working set size of 2 TiB.
3. Large Sequential I/O Size Test (1 MiB)
TEST COMMAND:
mpirun -mca btl tcp -np 128 -H hyper01:16,hyper02:16,hyper03:16,hyper04:16,hyper05:16,hyper06:16,hyper07:16,hyper08:16 \
/stratistor/modules/perftools/ior/ior -t 1m -b 256m -s 64 -F -C -e -i 2 -o /stratistor/clustermounts/machines/ior-test/testfileExplanation:
Executes the IOR tool across eight (8) test nodes with a total of 128 tasks, 16 clients per node, using 64 segments, with a transfer size of 1 MiB, block size of 256 MiB, and a total aggregate working set size of 2 TiB.
4. Small Random I/O Size Test (4 KiB)
TEST COMMAND:
mpirun -mca btl tcp -np 128 -H hyper01:16,hyper02:16,hyper03:16,hyper04:16,hyper05:16,hyper06:16,hyper07:16,hyper08:16 \
/stratistor/modules/perftools/ior/ior -t 4k -b 256m -s 64 -F -C -e -i 2 -z -o /stratistor/clustermounts/machines/ior-test/testfileExplanation:
Executes the IOR tool across eight (8) test nodes with a total of 128 tasks, 16 clients per node, using 64 segments, with a transfer size of 4 KiB, block size of 256 MiB, and a total aggregate working set size of 2 TiB.
5. Medium Random I/O Size Test (32 KiB)
TEST COMMAND:
mpirun -mca btl tcp -np 128 -H hyper01:16,hyper02:16,hyper03:16,hyper04:16,hyper05:16,hyper06:16,hyper07:16,hyper08:16 \
/stratistor/modules/perftools/ior/ior -t 32k -b 256m -s 64 -F -C -e -i 2 -z -o /stratistor/clustermounts/machines/ior-test/testfileExplanation:
Executes the IOR tool across eight (8) test nodes with a total of 128 tasks, 16 clients per node, using 64 segments, with a transfer size of 32 KiB, block size of 256 MiB, and a total aggregate working set size of 2 TiB.
6. Large Random I/O Size Test (1 MiB)
TEST COMMAND:
mpirun -mca btl tcp -np 128 -H hyper01:16,hyper02:16,hyper03:16,hyper04:16,hyper05:16,hyper06:16,hyper07:16,hyper08:16 \
/stratistor/modules/perftools/ior/ior -t 1m -b 256m -s 64 -F -C -e -i 2 -z -o /stratistor/clustermounts/machines/ior-test/testfileExplanation:
Executes the IOR tool across eight (8) test nodes with a total of 128 tasks, 16 clients per node, using 64 segments, with a transfer size of 1 MiB, block size of 256 MiB, and a total aggregate working set size of 2 TiB.
¶ FIO
1. Small Sequential I/O Size Test (4 KiB)
TEST COMMAND:
fio --filename=/stratistor/clustermounts/machines/fio/fiotest --ioengine=libaio --time_based --group_reporting --refill_buffers --norandommap \
--randrepeat=0 --unlink=1 --name=write-random-write --nrfiles=1 --direct=1 --iodepth=64 --runtime=10 --numjobs=8 --rw=write --bs=4K --size=32G \
--startdelay=2 --output-format=jsonExplanation:
Executes the FIO tool on a single host using 4 KiB block sizes, queue depth of 64 commands, cache disabled, 32GiB working data set.
2. Small Random I/O Size Test (4 KiB)
TEST COMMAND:
fio --filename=/stratistor/clustermounts/machines/fio/fiotest --ioengine=libaio --time_based --group_reporting --refill_buffers --norandommap \
--randrepeat=0 --unlink=1 --name=write-random-write --nrfiles=1 --direct=1 --iodepth=64 --runtime=10 --numjobs=8 --rw=randwrite --bs=4K --size=32G \
--startdelay=2 --output-format=jsonExplanation:
Executes the FIO tool on a single host using 4 KiB block sizes, queue depth of 64 commands, cache disabled, 32GiB working data set.
¶ S3 WARP
CLIENT EXECUTION
Command on each test client node:
warp clientSERVER EXECUTION
Command on each test server node:
warp get --duration=3m --warp-client=10.184.25.73,10.184.27.23,10.184.19.151,10.184.18.81 --host=192.168.101.1:15443,192.168.101.2:15443,192.168.101.3:15443 \
--access-key=[S3 ACCESS KEY] --secret-key=[S3 SECRET KEY] --tls --bucket bucket-001 --insecure¶ PERFORMANCE RESULTS
¶ TEST SYSTEM HARDWARE
The following test results are derived from a high-performance HyperSERV Hyper-Converged Infrastructure (HCI) deployment, engineered to deliver outstanding compute and storage performance in a compact, scalable design. This setup highlights the power of next-generation hardware components seamlessly integrated with StratiSYSTEM OS—purpose-built and optimized for demanding enterprise and data center workloads. The software architecture ensures consistent, linear scalability as additional nodes are introduced.
Each node in the cluster consists of a Supermicro SYS-221BT-HNC8R server equipped with:
2× Intel® Xeon® Gold 6438Y+ CPUs - 64 physical cores and 128 threads per node @2.0 GHz, offering a balanced mix of performance and power efficiency for multi-threaded workloads.
256 GB of DDR4 RAM, ensuring sufficient memory capacity to support virtualization, in-memory processing, and high-throughput tasks.
6× Samsung 3.84TB PCIe Gen5 NVMe SSDs (Model: MZWLO3T8HCLS), delivering ultra-fast storage performance with high IOPS, low latency, and sustained throughput, ideal for storage-intensive applications.
1× Mellanox ConnectX-7 400GbE network adapter (MT2910), providing extremely high network bandwidth and low-latency connectivity, enabling fast data movement across nodes and to external clients.
SteelDome StratiSYSTEM OS 3.3, tailored for hyper-converged deployments with features focused on performance, reliability, and ease of management.
¶ IOR TEST RESULTS (4-NODE HCI)
TEST COMMAND:
mpirun -mca btl tcp -np 256 -H hyper01:64,hyper02:64,hyper03:64,hyper04:64 /stratistor/modules/perftools/ior/ior \
-t 4k -b 256m -s 256 -F -C -e -u -o /stratistor/clustermounts/ior-test/testfileIOR 4-NODE TEST RESULTS:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB)
------ --------- ---------- ---------- ---------- ---------
write 14,862 3,804,719 0.000067 256 4
read 39,161 10,029,039 0.000017 256 4
¶ IOR TEST RESULTS (8-NODE HCI)
TEST COMMAND:
mpirun -mca btl tcp -np 256 -H hyper01:32,hyper02:32,hyper03:32,hyper04:32,hyper05:32,hyper06:32,hyper07:32,hyper08:32 /stratistor/modules/perftools/ior/ior \
-t 4k -b 256m -s 256 -F -C -e -u -o /stratistor/clustermounts/ior-test/testfileIOR 8-NODE TEST RESULTS:
access bw(MiB/s) IOPS Latency(s) block(KiB) xfer(KiB)
------ --------- ---------- ---------- ---------- ---------
write 29,923 7,660,475 0.000017 256 4
read 94,769 24,262,194 0.000005 256 4
¶ FRAMETEST RESULTS (8-NODE HCI)
To evaluate system performance at scale, Frametest was executed concurrently across eight HyperSERV nodes, using 4K frames to reflect real-world demands in media and entertainment environments. This approach aligns with industry standards for testing high-throughput, low-latency infrastructure—ensuring the platform can reliably handle the intense concurrent data and graphics processing requirements typical of video rendering, post-production, and live content delivery pipelines.
WRITE TEST COMMANDS:
(nodes 1-8): frametest -w 4k -n5000 -t20 .FRAMETEST WRITE THROUGHPUT TEST RESULTS:
NODE # Data Rate Framerate
------ ------------- ------------
1 5443.49 MB/s 111.8 fps
2 5525.70 MB/s 113.5 fps
3 5475.75 MB/s 112.5 fps
4 5559.42 MB/s 114.2 fps
5 5611.81 MB/s 115.3 fps
6 5492.07 MB/s 112.8 fps
7 5498.72 MB/s 112.9 fps
8 5603.56 MB/s 115.1 fps
-----------------------------------
TOTAL: 48,709.5 MB/s 1008.1 fps
READ TEST COMMANDS:
(nodes 1-8): frametest -r 4k -n5000 -t20 .FRAMETEST READ THROUGHPUT TEST RESULTS:
NODE # Data Rate Framerate
------ ------------- -----------
1 13343.73 MB/s 274.1 fps
2 13290.64 MB/s 273.0 fps
3 13333.80 MB/s 273.9 fps
4 13343.54 MB/s 274.1 fps
5 13337.88 MB/s 273.9 fps
6 13336.59 MB/s 273.9 fps
7 13298.43 MB/s 273.2 fps
8 13441.37 MB/s 274.0 fps
----------------------------------
TOTAL: 106,727 MB/s 2190.1 fps
¶ FIO TEST RESULTS (SINGLE VM INSTANCE - DIRECT I/O CACHE BYPASS)
The following test results are based on a virtual machine deployed directly on the HyperSERV platform, utilizing the high-performance hardware configuration detailed above. This setup is designed to emulate real-world operating conditions, ensuring that the results reflect the true capabilities of the system in enterprise-grade environments.
FIO IOPS TEST COMMANDS:
fio --rw=read --refill_buffers --norandommap --randrepeat=0 --ioengine=io_uring --bs=4k --iodepth=128 --numjobs=12 --size=128g --time_based --runtime=60 \
--group_reporting --filename=/mnt/vdb/fio-test --name=jobs-test-job --direct=1 --cpus_allowed_policy=split
fio --rw=randread --refill_buffers --norandommap --randrepeat=0 --ioengine=io_uring --bs=4k --iodepth=128 --numjobs=12 --size=128g --time_based --runtime=60 \
--group_reporting --filename=/mnt/vdb/fio-test --name=jobs-test-job --direct=1 --cpus_allowed_policy=split
fio --rw=write --refill_buffers --norandommap --randrepeat=0 --ioengine=io_uring --bs=4k --iodepth=128 --numjobs=12 --size=128g --time_based --runtime=60 \
--group_reporting --filename=/mnt/vdb/fio-test --name=jobs-test-job --direct=1 --cpus_allowed_policy=split
fio --rw=randwrite --refill_buffers --norandommap --randrepeat=0 --ioengine=io_uring --bs=4k --iodepth=128 --numjobs=12 --size=128g --time_based --runtime=60 \
--group_reporting --filename=/mnt/vdb/fio-test --name=jobs-test-job --direct=1 --cpus_allowed_policy=splitFIO IOPS TEST RESULTS:
Test Type Operation Block Size IOPS Bandwidth
----------- ---------- ---------- ----- ----------
IOPS Test 1 Seq Read 4K 815K 3.18 GiB/s
IOPS Test 2 Rand Read 4K 100K 391 MiB/s
IOPS Test 3 Seq Write 4K 662K 2.59 GiB/s
IOPS Test 4 Rand Write 4K 79.1K 309 MiB/s
FIO THROUGHPUT TEST COMMANDS:
fio --rw=read --refill_buffers --norandommap --randrepeat=0 --ioengine=io_uring --bs=1m --iodepth=32 --numjobs=4 --size=128g --time_based --runtime=60 \
--group_reporting --filename=/mnt/vdb/fio-test --name=jobs-test-job --direct=1 --cpus_allowed_policy=split
fio --rw=randread --refill_buffers --norandommap --randrepeat=0 --ioengine=io_uring --bs=1m --iodepth=32 --numjobs=4 --size=128g --time_based --runtime=60 \
--group_reporting --filename=/mnt/vdb/fio-test --name=jobs-test-job --direct=1 --cpus_allowed_policy=split
fio --rw=write --refill_buffers --norandommap --randrepeat=0 --ioengine=io_uring --bs=1m --iodepth=32 --numjobs=4 --size=128g --time_based --runtime=60 \
--group_reporting --filename=/mnt/vdb/fio-test --name=jobs-test-job --direct=1 --cpus_allowed_policy=split
fio --rw=randwrite --refill_buffers --norandommap --randrepeat=0 --ioengine=io_uring --bs=1m --iodepth=32 --numjobs=4 --size=128g --time_based --runtime=60 \
--group_reporting --filename=/mnt/vdb/fio-test --name=jobs-test-job --direct=1 --cpus_allowed_policy=splitFIO THROUGHPUT TEST RESULTS:
Test Type Operation Block Size IOPS Bandwidth
----------- ---------- ---------- ----- ----------
Throughput 1 Seq Read 1M 20.3K 19.9 GiB/s
Throughput 2 Rand Read 1M 8039 8.04 GiB/s
Throughput 3 Seq Write 1M 3460 3.46 GiB/s
Throughput 4 Rand Write 1M 1870 1.87 GiB/s
¶ CONCLUSION
By adhering to this comprehensive testing framework, organizations can effectively utilize tools such as Perfprofiler, IOR, FIO, Frametest, and S3 Warp to evaluate and enhance storage system performance. This structured methodology ensures that testing is conducted in a consistent and repeatable manner, delivering results that are both reliable and relevant to real-world use cases—including high-performance computing (HPC), media production, cloud storage, hyper-converged infrastructures, and tiered over-the-wire deployments.