What Is Object Storage?
Object-based storage is a method of managing large data stores on a variety storage media and shared storage resources. While it has some similarities to file storage, the differences between the two access and management protocols are significant.
Object storage has been around for a while -- some of its key attributes were defined by Seagate in 1999 -- but it has only become a viable storage option in the last 15 years or so. Like file storage, object storage is used principally for unstructured data, which it manages as discreet chunks that can be of any size and contain various types of data, such as text, graphics or audio/video media.
Access to object storage data is accomplished using the REST (representational state transfer) APIs over HTTP, versus the POSIX (Portable Operating System Interface) standards used by file storage systems. REST is a simple and lightweight method of transferring data over the web, so it’s a low-overhead, easy-to-use method of accessing object storage data.
Object-based storage is used by most commercial cloud storage products, such as AWS’s S3 (Simple Storage Service) cloud storage service.
How Does Object Storage Work?
Object storage’s key feature is the way it keeps track of the data objects it manages. It uses a flat file database as its directory and management system, which has proven to be much more expandable than file storage’s hierarchical system of nested folders and files.
The database retains a unique identifier for each object. The 64-bit Object ID (OID) indicates the location of the object on a single storage medium or among a cluster of storage devices. Unlike block storage, which allocates storage in predefined blocks of equal length, the lengths of objects can vary. As noted, the relatively simple system of keeping track of objects makes it possible to extend a single object storage system across multiple storage resources. A file storage system, on the other hand, has a defined limit on the number of files it can manage. While some NAS file systems may be quite large, they generally can’t expand to the degree that object storage can.
Another distinguishing characteristic of object storage is the way it handles metadata related to each stored object. A file system -- like the Windows file directory on a PC or a shared NAS system -- includes some basic metadata related to each file it manages, such as file name, file size, date created, date modified and possibly the application it’s associated with. The file system metadata is maintained in the directory that also holds information about the location of the file’s components. Object storage also stores metadata about each object, but it bundles that information with the object itself so that it doesn’t add to the size of the directory. But the most important difference comparing object storage metadata with file storage metadata is that it is customizable. Users can add their own metadata to objects to make them easier to identify or to give them useful attributes.
Another key difference between object storage and file storage is the way the two systems handle data protection. Object storage uses erasure coding rather than RAID to protect data. While most file storage offers several levels of RAID to choose from, the more protection required typically translates to more storage overhead required. Erasure coding is like RAID in that it can parse an object in multiple parts and store those parts in different locations. Erasure coding tends to use less storage capacity to provide an equivalent (or near equivalent) level of protection. With erasure coding, you can effective “dial up” the desired level of protection by deciding how many parts an object should be broken into and where the parts should reside.
What Are The Benefits of Object Storage?
Object storage’s expandability makes it good choice for applications that require massive amounts of data, such as large archives, data warehouses and data lakes. And because a single object system can spread across multiple hardware systems, future expansion is also manageable.
The sheer number of data objects that can be managed in a single system distributed among many devices makes object an ideal choice for cloud storage providers. In addition, because objects can be very large files, an object storage system is also ideal for streaming media applications.
The ability to customize metadata for objects makes it easy to provide much more information about stored data than what file or block storage can offer. Customizable metadata can also be a key factor in automating storage management. For example, an “end of life” attribute might be added to objects that would automatically trigger their deletion or archiving when they reached a certain age. Additionally, metadata can be used to strengthen security by making objects accessible only to users with certain attributes that match the defined metadata.
Object storage’s simplicity, facility for handling vast numbers of objects and capability to store very large files using off-the-shelf hardware make it a very low-cost storage option -- often much less expensive than file or block storage with similar capacity requirements.
What Are The Drawbacks of Object Storage?
Object storage does have a few drawbacks, such as its performance, which generally doesn’t match that of block or even file storage. With a flat file directory system that might be expansive, the system is slower compared with file or block when locating data. The diminished performance is essentially the trade-off for a storage system with excellent expandability and ability to handle files of varying sizes. That’s not to say that object storage couldn’t be used for structured-content applications like databases. It just wouldn’t perform well for large or very active databases.
Another factor contributing to object storage’s performance is the method it uses to update existing data. While file or block storage allow users to modify existing data segments, object storage doesn’t do in-place updates. Rather, object storage replaces the entire object when it is modified. This creates slightly more operational overhead, which also degrades performance.
Perhaps the biggest drawback to object storage is the inability for many applications to access object storage natively. Most office productivity applications use the familiar file storage system to store and access data. Structured data applications expect to retrieve uniformly sized blocks of data for their calculations and data manipulations. Object storage can be used as a backend storage repository for those types of applications, but it would require emulation or protocol translation software to make it look like file or block to appropriate applications. That’s generally how cloud storage providers are able to package their object-based storage as file or block storage.
Use Cases for Object Storage
Given its benefits and shortcomings, object storage isn’t necessarily a general-purpose storage option for most companies. Its strengths make it a good choice for numerous use cases, however.
- Cloud storage services. With its software-defined approach to storage management, object storage is an ideal option for cloud storage because it’s expandable and can be “stretched” over multiple standard hardware systems. Perhaps the best know example is AWS’ S3 cloud storage service, but all major cloud providers use some form of object storage.
- Streaming media applications. Because it can easily accommodate large files of varying sizes, object storage works well for streaming media and other content distribution applications.
- Archive or backup. With nearly unlimited capacity, object storage is ideal for storing data that will be accessed infrequently, such as backup datasets or archival data. The backup or archiving software must support object as target or work with some form of emulation.
- IoT implementations. IoT applications, particularly those used for video surveillance and industrial sensor monitoring, tend to accumulate large amounts of data. Object’s configurable metadata helps to keep track of and manage that specialized data and enable real-time analytics on large data volumes.
- Low-cost, expandable replacement for file storage. Assuming performance isn’t an issue, object storage coupled with a file system frontend, such as NFS or SMB, could replace a NAS system, providing greater capacity flexibility to accommodate growth.
Conclusion: Object Adds a Choice Beyond File and Block
Object storage is a mature technology that has been tested and proven durable enough to serve as the foundation of some of the largest cloud storage installations in the world. Its relative simplicity, customization features and vast capacities make it a solid alternative -- or effective complement -- to file or block storage.
