SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding. Enterprise version is at seaweedfs.com.

Partisan is a scalable and flexible, TCP-based membership system and distribution layer for the BEAM. It bypasses the use of Distributed Erlang for manual connection management via TCP, and has several pluggable backends for different deployment scenarios.

Partisan is a runtime system that enables greater scalability and reduced latency for distributed actor applications.

• Partisan improves scalability by allowing the application developer to specialize the overlay network to the application’s communication patterns.
• Partisan achieves lower latency by leveraging several predominately automatic optimizations that result in the efficient scheduling of messages.
• Bring Your Own Overlay - Partisan exposes an API for users to implement their own overlays; application developers must simply implement the membership_strategy interface for handling messages. Partisan automatically uses this membership strategy for processing incoming and outgoing messages to the system – the application developer only needs to handle internal state transitions and supplying the system with an updated list of members. Partisan automatically sets up required connections, serializes and deserializes messages, performs failure detection, and message forwarding. This makes it possible to implement protocols with very little code; our implementation of the full-mesh membership protocol is 152 LOC.
• Partisan is the first distributed actor system to expose this level of control to the application developer, improving the performance of existing actor application and enabling new types of actor applications.

An open-source distributed object storage service tailored for self-hosting

In 2020, I presented a talk at FrOSCon titled No, we won’t have a video call for that: Communications for distributed teams. In early 2021 I put together a full-length writeup of that talk, and published it here. And for no apparent reason that article then made it to the top of Hacker News on one day in September 2021,1 and apparently resonated with quite a few people.

And after that, I got a fair number of questions along the lines of “okay, what you talk about is a spot-on description of how I want to work, but how do I get there?” In other words, what can you do in order to transition from a team that’s stuck in meeting hell, to one that actually goes fully distributed and embraces asynchronous communications?

Note: I use the shorthand Meeting Hell for a situation in which people are forced to
be in unnecessary and unproductive video meetings2 for an unhealthy fraction of
their work time.

This is admittedly a mere symptom of not having adopted asynchronous and
distributed ways of working, but it’s such a tell-tale sign thereof that it counts as a
dead giveaway. Thus, I think it’s okay to say “I’m stuck in meeting hell” when what
someone means is really “I work in an organization that’s failing badly at
distributed and asynchronous work.”

After No, We Won’t Have a Video Call for That, which covered how productive distributed teams operate, and the Getting out of Meeting Hell series, which focused on how you as an individual can get to being a member of a functional distributed team, let’s zoom out a bit.

Let’s discuss a slightly larger, organizational picture, just in case you ask yourself this question: “Why isn’t my organization distributed and asynchronous? And, as companies bleed talent right and left because competent people run for the competition that does get distributed work right, why don’t they wake up and become like that? Can someone please make my company more distributed and asynchronous?”

For additional context, let me interject the following observations from Kris Köhntopp, who posted them in German on Twitter (I am taking the liberty to translate here; emphasis mine):

What’s funny is that [what matters to being successful as a distributed company]
are all learnable skills — written communications, sensible meeting prep and
follow-up, correct definition of objectives and tasks, etc.

That’s a craft.

But it appears that organizations prefer to bleed their teams dry by attrition, rather
than to learn, or to hire people that can help teams to write things down,
professionally maintain a wiki, and teach and drive communications.

And further downthread, Kris says (again, my translation and emphasis):

This is all definitely feasible, after all there’s remote-first companies of substantial
size and they work.

They must have built themselves somehow, they didn’t get to where they are by
pure chance.

I agree with all of that (obviously), but things are complicated. So let’s drill into this a bit.

Mitogen is a Python library for writing distributed self-replicating programs.

There is no requirement for installing packages, copying files around, writing shell snippets, upfront configuration, or providing any secondary link to a remote machine aside from an SSH connection. Due to its origins for use in managing potentially damaged infrastructure, the remote machine need not even have free disk space or a writeable filesystem.

It is not intended as a generic RPC framework; the goal is to provide a robust and efficient low-level API on which tools like Salt, Ansible, or Fabric can be built, and while the API is quite friendly and comparable to Fabric, ultimately it is not intended for direct use by consumer software.

The focus is to centralize and perfect the intricate dance required to run Python code safely and efficiently on a remote machine, while avoiding temporary files or large chunks of error-prone shell scripts, and supporting common privilege escalation techniques like sudo, potentially in combination with exotic connection methods such as WMI, telnet, or console-over-IPMI.

Longhorn implements distributed block storage using containers and microservices. Longhorn creates a dedicated storage controller for each block device volume and synchronously replicates the volume across multiple replicas stored on multiple nodes. The storage controller and replicas are themselves orchestrated using Kubernetes.
Features

• Enterprise-grade distributed block storage with no single point of failure
• Incremental snapshot of block storage
• Backup to secondary storage (NFS or S3-compatible object storage) built on efficient change block detection
• Recurring snapshots and backups
• Automated, non-disruptive upgrades. You can upgrade the entire Longhorn software stack without disrupting running storage volumes.
• An intuitive GUI dashboard

In the past, ITOps and DevOps have found it hard to add replicated storage to Kubernetes clusters. As a result many non-cloud-hosted Kubernetes clusters don’t support persistent storage. External storage arrays are non-portable and can be extremely expensive.

Longhorn delivers simplified, easy to deploy and upgrade, 100% open source, cloud-native persistent block storage without the cost overhead of open core or proprietary alternatives.

Apache Pulsar is an open-source distributed pub-sub messaging system originally created at Yahoo and now part of the Apache Software Foundation

The Tooz project aims at centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications.

A single distribution of libraries that automatically collects traces and metrics from your app, displays them locally, and sends them to any analysis tool.

The key features of OpenCensus include:

• Standard wire protocols and consistent APIs for handling trace and metric data.
• A single set of libraries for many languages, including Java, C++, Go, .Net, Python, PHP, Node.js, Erlang, and Ruby.
• Included integrations with web and RPC frameworks, making traces and metrics available out of the box.
• Included exporters for storage and analysis tools. Right now the list includes Zipkin, Prometheus, Datadog, Stackdriver, and Azure App Insights.
• Full open source availability for additional integrations and export options.
• No additional server or daemon is required to support OpenCensus.
• In process debugging: an optional agent for displaying request and metrics data on instrumented hosts.

Fossil is a simple, high-reliability, distributed software configuration management system with these advanced features:

• Integrated Bug Tracking, Wiki, and Technotes - In addition to doing distributed version control like Git and Mercurial, Fossil also supports bug tracking, wiki, and technotes.

• Built-in Web Interface - Fossil has a built-in and intuitive web interface with a rich assortment of information pages (examples) designed to promote situational awareness.

• This entire website¹ is just a running instance of Fossil. The pages you see here are all wiki or embedded documentation. When you clone Fossil from one of its self-hosting repositories, you get more than just source code - you get this entire website. (¹except the download page)

• Self-Contained - Fossil is a single self-contained stand-alone executable. To install, simply download a precompiled binary for Linux, Mac, OpenBSD, or Windows and put it on your $PATH. Easy-to-compile source code is also available.

• Simple Networking - No custom protocols or TCP ports. Fossil uses ordinary HTTP (or HTTPS or SSH) for network communications, so it works fine from behind restrictive firewalls, including proxies. The protocol is bandwidth efficient to the point that Fossil can be used comfortably over dial-up.

• CGI/SCGI Enabled - No server is required, but if you want to set one up, Fossil supports four easy server configurations.

• Autosync - Fossil supports "autosync" mode which helps to keep projects moving forward by reducing the amount of needless forking and merging often associated with distributed projects.

• Robust & Reliable - Fossil stores content using an enduring file format in an SQLite database so that transactions are atomic even if interrupted by a power loss or system crash. Automatic self-checks verify that all aspects of the repository are consistent prior to each commit.

• Free and Open-Source - Uses the 2-clause BSD license.

The KV endpoint is used to access Consul's simple key/value store, useful for storing service configuration or other metadata.

We've started our first agent and registered and queried a service on that agent. This showed how easy it is to use Consul but didn't show how this could be extended to a scalable, production-grade service discovery infrastructure. In this step, we'll create our first real cluster with multiple members.

When a Consul agent is started, it begins without knowledge of any other node: it is an isolated cluster of one. To learn about other cluster members, the agent must join an existing cluster. To join an existing cluster, it only needs to know about a single existing member. After it joins, the agent will gossip with this member and quickly discover the other members in the cluster. A Consul agent can join any other agent, not just agents in server mode.

Developing Distributed Applications that use ZooKeeper

Apache ZooKeeper is the coolest technology I recently came across. I found it when I was doing a research about Solr Cloud features. I got very impressed by Solr’s distributed computing. You literately have to fire a new instance and it will automatically find its place in “the cloud”. It will assign itself to a particular shards and it will make a decision to become a leader or a replica. Later you can query any of the available servers and it will find you all required data even if it’s not on that server. If some of the servers fail the service will continue to work. Very dynamic, very clever, very cool.

TANGO is an object oriented distributed control system using CORBA (synchronous and asynchronous communication) and zeromq (event based communication) and is being actively developed as a collaborative effort between the following research institutes: Alba,Anka, Desy, Elettra, ESRF, FRM II, MAX-IV, Solaris, Soleil, ELI Beams, ELI Alps