SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.
cowsql (/ˈkaʊ,siːkwəl/ listen) is a C library that implements an embeddable and replicated SQL database engine with high availability and automatic failover.
cowsql extends SQLite with a network protocol that can connect together various instances of your application and have them act as a highly-available cluster, with no dependency on external databases.
The name "cowsql" loosely refers to the "pets vs. cattle" concept, since it's generaly fine to delete or rebuild a particular node of an application that uses cowsql for data storage.
NooBaa is a highly customizable and dynamic data gateway for objects, providing data services such as caching, tiering, mirroring, dedup, encryption, compression, over any storage resource including S3, GCS, Azure Blob, Filesystems, etc.
The goal is to simplify data flows for admins by connecting to any of the storage silos from private or public clouds, and providing a single scalable data services, using the same S3 API and management tools. NooBaa allows full control over data placement with dynamic policies per bucket or account.
The CERN Tape Archive (CTA) is the archival storage system for the custodial copy of all physics data at CERN. CTA is implemented as the tape back-end to EOS. EOS+CTA is the CERN archival solution for LHC Run–3 and beyond.
For a general overview of CTA, see our Publications and Presentations. There is also some general Development and Systems Integration Documentation which complements what is on this site.
EOS is an open-source storage software solution to manage Exabyte storage for the CERN Large Hadron Collidor LHC. Core of the implementation is the XRootD framework providing a feature-rich remote access protocol.
An open-source distributed object storage service tailored for self-hosting
Production ready management for File, Block and Object Storage
When you insert data into a database and run COMMIT you expect things to be there: Atomically, Consistent, Isolated and Durable, like Codd commanded us 40 years ago, but also quickly. There is a surprising amount of sophistication being poured into this, but since I do not want to shame MongoDB and Redis developers in this post, I am not going to talk about that much in this place.
We are instead trying to understand what our databases are doing all day, from the point of view of the storage stack.
Longhorn implements distributed block storage using containers and microservices. Longhorn creates a dedicated storage controller for each block device volume and synchronously replicates the volume across multiple replicas stored on multiple nodes. The storage controller and replicas are themselves orchestrated using Kubernetes.
Features
- Enterprise-grade distributed block storage with no single point of failure
- Incremental snapshot of block storage
- Backup to secondary storage (NFS or S3-compatible object storage) built on efficient change block detection
- Recurring snapshots and backups
- Automated, non-disruptive upgrades. You can upgrade the entire Longhorn software stack without disrupting running storage volumes.
- An intuitive GUI dashboard
In the past, ITOps and DevOps have found it hard to add replicated storage to Kubernetes clusters. As a result many non-cloud-hosted Kubernetes clusters don’t support persistent storage. External storage arrays are non-portable and can be extremely expensive.
Longhorn delivers simplified, easy to deploy and upgrade, 100% open source, cloud-native persistent block storage without the cost overhead of open core or proprietary alternatives.
Velero is a backup and restore solution for Kubernetes, that can be used to take backups and restore them in case of loss but also for migrating to other clusters.
Authors:
Jie Yu <jie@mesosphere.io> (@jieyu)
Saad Ali <saadali@google.com> (@saad-ali)
James DeFelice <james@mesosphere.io> (@jdef)
container-storage-interface-working-group@googlegroups.comThe Kubernetes implementation of the Container Storage Interface (CSI) has been promoted to GA in the Kubernetes v1.13 release. Support for CSI was introduced as alpha in Kubernetes v1.9 release, and promoted to beta in the Kubernetes v1.10 release.
The GA milestone indicates that Kubernetes users may depend on the feature and its API without fear of backwards incompatible changes in future causing regressions. GA features are protected by the Kubernetes deprecation policy.
blkreplay is a GPL’ed userspace toolkit, driving the block layer of Linux (or other Unix-like OSes) while measuring latency and throughput of IO operations for later visualization (so-called “sonar diagrams” and others).
In addition to artificial loads like random read-write sweeps and various kinds of overload tests, blkreplay can replay natural loads which have been recorded by blktrace (or Windows DiskMon) at heavily-loaded production servers at big data centers.
Swift is a highly available, distributed, eventually consistent object/blob store. Organizations can use Swift to store lots of data efficiently, safely, and cheaply.
This documentation is generated by the Sphinx toolkit and lives in the source tree. Additional documentation on Swift and other components of OpenStack can be found on the OpenStack wiki and at http://docs.openstack.org.
S3 is a product from Amazon, and as such, it includes “features” that are outside the scope of Swift itself. For example, Swift doesn’t have anything to do with billing, whereas S3 buckets can be tied to Amazon’s billing system. Similarly, log delivery is a service outside of Swift. It’s entirely possible for a Swift deployment to provide that functionality, but it is not part of Swift itself. Likewise, a Swift deployment can provide similar geographic availability as S3, but this is tied to the deployer’s willingness to build the infrastructure and support systems to do so.