Listing Objects
QStorage provides several ways to list the objects in your buckets. You can list all objects, list objects with a specific prefix, or list objects hierarchically using delimiters. This topic explains the different methods for listing objects in QStorage.
Understanding Object Listing
When you list objects in a bucket, QStorage returns a list of objects up to 1,000 objects per response by default. You can use pagination parameters to retrieve a subset of objects or to iterate through the complete list of objects.
Important Concepts for Listing Objects
- Prefix: Filter the results to include only objects whose keys begin with the specified prefix.
- Delimiter: Character used to group keys. If you specify a delimiter, the operation returns a list of distinct key prefixes in
CommonPrefixes
. These prefixes represent keys that share a common prefix up to the delimiter. - Pagination: For buckets with many objects, the list operation might return only a portion of the objects. In these cases, the response includes a continuation token that you can use in a subsequent request to fetch the next set of objects.
Listing All Objects in a Bucket
- Using QConsole
- Using Q's CLI Tooling
- Using a Third-Party S3-Compatible CLI
To list all objects in a bucket using QConsole:
- Sign in to QConsole
- Navigate to QStorage
- Select the bucket you want to view
- The objects in the bucket will be displayed in the console
To list all objects in a bucket using Q's CLI:
q s3 ls s3://bucket-name
For a more detailed listing:
q s3 ls s3://bucket-name --recursive
If you're using a third-party S3-compatible CLI, you can list objects with:
s3cmd ls s3://bucket-name
Or with the AWS CLI:
aws s3 ls s3://bucket-name --endpoint-url https://qstorage.quilibrium.com
Listing Objects with a Prefix
To list objects with a specific prefix (for example, all objects in a "folder"):
- Using QConsole
- Using Q's CLI Tooling
- Using a Third-Party S3-Compatible CLI
- Sign in to QConsole
- Navigate to QStorage
- Select the bucket you want to view
- Navigate to the folder or use the search functionality to filter by prefix
q s3 ls s3://bucket-name/prefix/
s3cmd ls s3://bucket-name/prefix/
Or with the AWS CLI:
aws s3 ls s3://bucket-name/prefix/ --endpoint-url https://qstorage.quilibrium.com
Listing Objects Hierarchically (Using Delimiters)
When you use a delimiter, QStorage treats the delimiter as a break in the hierarchy, allowing you to list objects in a folder-like structure.
- Using QConsole
- Using Q's CLI Tooling
- Using a Third-Party S3-Compatible CLI
QConsole automatically uses delimiters to present objects in a hierarchical structure.
q s3 ls s3://bucket-name --delimiter "/"
With the AWS CLI:
aws s3api list-objects-v2 --bucket bucket-name --delimiter "/" --endpoint-url https://qstorage.quilibrium.com
Pagination for Large Object Lists
For buckets with many objects, you'll need to handle pagination:
- Using Q's CLI Tooling
- Using a Third-Party S3-Compatible API
The CLI tools handle pagination automatically in most cases.
aws s3api list-objects-v2 --bucket bucket-name --max-items 100 --starting-token YOUR_CONTINUATION_TOKEN --endpoint-url https://qstorage.quilibrium.com
Best Practices for Listing Objects
- Use prefixes and delimiters to organize and retrieve objects efficiently
- Implement pagination when dealing with buckets containing many objects
- Consider performance implications when listing large buckets frequently
- Use appropriate listing patterns based on your application's needs:
- For simple browsing, use delimiter-based listing
- For complete inventory, use recursive listing
- For specific file types, use prefix filtering
Performance Considerations
Listing operations on buckets with a large number of objects can be resource-intensive. Consider the following:
- Avoid frequent listing of entire buckets with many objects
- Use specific prefixes to narrow down the results
- Implement caching mechanisms if your application frequently lists the same objects
- For applications that need to track changes, consider using bucket notifications instead of repeated listing operations