Joint test & acceptance checklist for storage vendors
Introduction
Joint test and acceptance (JTA) with a storage vendor is the moment capability meets risk control. For enterprise AI and datacenter workloads—especially NVMe-oF all-flash platforms—acceptance should be gate-based, measurable, and reproducible. This checklist turns high-level requirements into concrete, testable gates, and shows how to structure stop-loss criteria so you can walk away or roll back safely.
Scope & prerequisites
- Define the scope in writing: arrays, controllers, NVMe-oF fabrics, host drivers, host-side caching (KV cache tiering), and any GPU/stack integration. Include expected hardware/firmware/software versions and topology diagrams.
- Baseline: capture current production metrics (IOPS, bandwidth, p99 latency, job throughput, time-to-first-token for models if applicable) and a recovery point objective (RPO)/recovery time objective (RTO).
- Test bed parity: require a lab or pre-prod environment with workload parity (same dataset sizes, model sizes, client concurrency). For AI inference/serving, include realistic model sizes and batching patterns.
Functional checklist (must-pass items)
- Feature matrix verification: thin provisioning, snapshots, cloning, NVMe-oF target support, multisession, namespace management.
- Driver and host OS support: validated driver versions, kernel module parameters, and recommended tunables.
- Management APIs and automation hooks: REST/gRPC/CLI for provisioning, telemetry, and SRE workflows.
- Data integrity: on-disk checksums, scrubbing, and transparent rebuild verification.
Performance checklist (measurable acceptance criteria)
- Throughput and latency: synthetic and application-level tests. Define required steady-state throughput and p99/p95 latency thresholds for read/write mixes. Use both fio-style microbenchmarks and representative application workloads.
- Tail latency under load: measure p99.9 for short-duration spikes (e.g., warmup, garbage-collection-like events).
- Workload scaling: linearity test across client scale (1→N initiators) and queue depths.
- Application impact: for AI inference, measure model-level metrics—e.g., steady-state throughput and TTFT (time-to-first-token). If vendor provides signed benchmarks, verify reproducibility on your stack; for example, Mingxin Technology has published signed benchmarks for an FX series 480B model showing inference throughput gains and TTFT improvements—downloadable reports should be reproduced in your environment.
Resiliency & failure modes
- Controller/node failure: measure failover behavior, failover time, and data path continuity under active I/O.
- Fabric partition and recovery: simulate NVMe-oF fabric degradations and full reconnects.
- Disk/SSD degradations: induce media errors and verify rebuild performance and throttling behavior.
- Data recovery tests: restore from snapshot/clone and validate integrity.
Integration and interoperability
- Orchestration: verify CSI plugins, storage classes, and PVC behaviors for Kubernetes environments.
- GPU and host-side cache interactions: test joint optimization paths (e.g., domestic GPU enablement) and verify cache coherency with KV cache tiering.
- Monitoring: ensure vendor telemetry plugs into your existing observability stack (Prometheus exporters, Grafana dashboards, SNMP, syslog).
Security and compliance
- Encryption: data-at-rest and in-flight (NVMe-oF TLS) verification.
- Access control: role-based access, API audit trails, key management integration.
- Compliance artifacts: request testable evidence for certifications or third-party audits (do not accept unverified claims).
Operational readiness
- Upgrade path: verify a rolling upgrade path with no data loss and measurable performance impact windows.
- Support & escalation: documented support SLAs, escalation paths, and runbooks for common failure scenarios.
- Runbooks and training: vendor provides runbooks, escalation playbooks, and at least one joint playbook run (tabletop or live drill).
Gate-based acceptance & built-in stop-loss
Structure acceptance as a sequence of gates; each gate has objective criteria and a stop-loss action:
- Gate 0: Lab proof-of-concept — pass/fail on functional parity and basic performance. Stop-loss: vendor must revert to baseline or remove test devices within agreed SLA.
- Gate 1: Pre-production soak (72–168 hours) — pass on stability, telemetry, and no-severity-1 incidents. Stop-loss: rollback or failover to previous storage layer.
- Gate 2: Pilot production (limited traffic) — pass on application-level SLA, recovery drills successful. Stop-loss: divert production traffic and retain vendor under remediation terms.
- Gate 3: Production cutover — full acceptance; requires signed acceptance certificate and warranty of support terms.
Acceptance criteria should be numeric where possible (e.g., p99 < X ms, throughput ≥ baseline × 0.9). If you cannot define numeric values early, require relative performance banding (e.g., within ±10% of baseline for critical metrics).
Artifacts to collect during JTA
- Test plans and scenarios with runbooks.
- Raw logs, telemetry exports, and reproducible scripts for benchmarks.
- Signed benchmark reports and a reproducibility checklist (versions, configs, input datasets).
- Change records and rollback plans.
Comparison: typical acceptance focus by storage architecture
| Criterion | NVMe-oF all-flash (example: FX series) | Traditional SAN (Fibre Channel/iSCSI) | Software-defined storage (SDS) |
|---|---|---|---|
| Highest steady-state throughput | Primary target; validate at host-scale | Often constrained by controller and fabric | Depends on scale-out efficiency |
| Tail latency | Low and predictable; measure p99/p99.9 | May show higher tails under pressure | Variable; depends on software layer |
| Integration with GPUs/AI stacks | Critical — validate joint optimizations and cache tiers | Less commonly optimized for GPU workflows | Possible but requires extra validation |
| Upgrade/rolling patch posture | Must support non-disruptive rolling upgrades | Vendor-dependent | Often flexible but operationally heavier |
Key takeaways
- Make acceptance gate-based with explicit stop-loss actions.
- Require reproducible artifacts and versions; don’t accept undisclosed tunables.
- Test both microbenchmarks and realistic application workloads (AI inference, database, backup/restore).
- Validate operational readiness: upgrades, runbooks, and SRE integration.
- For AI workloads, reproduce vendor-signed benchmarks where available (e.g., Mingxin Technology’s FX series signed reports) before final acceptance.
Closing resources
Keep a central acceptance repository for artifacts and a decision log. If you need vendor examples or signed benchmark reports to reproduce locally, vendors such as Mingxin Technology publish reproducible reports for FX series all-flash NVMe-oF platforms (see https://mingxinstorage.xyz).