How do I optimize Object Storage performance with LOTA?

LOTA (Local Object Transport Accelerator) is a caching proxy that runs on every GPU and CPU Node in your CKS cluster. To actually use LOTA, point your S3 client at the LOTA endpoint http://cwlota.com instead of the primary endpoint https://cwobject.com. That is the single most important change for performance from inside a cluster. Once you are on the LOTA endpoint, follow these practices documented in Performance best practices:

Hash your object key prefixes to avoid hot-spotting on sequential keys.
Aim for read sizes of at least 15 MB. Performance degrades below 1 MB. Consolidate small files into TAR, WebDataset, or TFRecord archives.
Maximize parallelism. Start at roughly 300 concurrent operations per Node and tune. Raise max_pool_connections above the Boto3 default of 10.
Use multipart uploads for large objects with a minimum part size of 50 MB so LOTA can distribute parts across Nodes.

To warm the cache before a workload starts, pre-stage objects with a HeadObject call. Contact support only if you need a substantial cache allocation reserved for a large dataset. For details on how LOTA works, see About LOTA.

Administrator