In this post I will explain how to transfer a multi terabyte size database between two MySQL instances using mydumper, which is a logical backup and restore tool that works in parallel. I will also cover the case where you need to transfer only a subset of data.
Big tables are often problematic because of the amount of time needed to do mysqldump/restore which is single threaded. It is not uncommon for this type of process to take several days.
It’s been 3 months since I first wrote about our efforts to scale Git to extremely large projects and teams with an effort we called “Git Virtual File System”. As a reminder, GVFS, together with a set of enhancements to Git, enables Git to scale to VERY large repos by virtualizing both the .git folder and the working directory. Rather than download the entire repo and checkout all the files, it dynamically downloads only the portions you need based on what you use.
Low bug count, good performance, easy modification. Good code is high-impact, and is perhaps the main reason behind the existence of the proverbial 10x developer. And yet, despite it’s importance, it eludes new developers. Literature on the subject usually amounts to disconnected collections of tips. How can a new developer just memorize all that stuff? “Code Complete“, the greatest exponent in this matter, is 960 pages long!
I believe it’s possible to construct a simple mental framework that can be used with any language or library and which will lead to good quality code by default. There are five main concepts I will talk about here. Keep them in mind and writing good code should be a breeze.
Update: Mia Li was kind enough to offer a Chinese translation of this post here.
Once a packet arrives as a particular CloudFlare data center we want to ensure it gets to a server that can correctly handle the request. There are four key tasks that CloudFlare's servers perform: DNS, proxy, cache, and logging. We tend to follow the Google-like approach and deploy generic, white-box servers that can perform a number of different functions. (Incidentally, if anyone is interested, we're thinking of doing a blog post to "tour" a typical CloudFlare server and discuss the choices we made in working with manufacturers to design them.) Since servers can fail or be overloaded, we need to be able to route traffic intelligently around problems. For that, we return to our old friend Anycast.
BPF, as in Berkeley Packet Filter, was initially conceived in 1992 so as to provide a way to filter packets and to avoid useless packet copies from kernel to userspace. It initially consisted in a simple bytecode that is injected from userspace into the kernel, where it is checked by a verifier—to prevent kernel crashes or security issues—and attached to a socket, then run on each received packet. It was ported to Linux a couple of years later, and used for a small of applications (tcpdump for example). The simplicity of the language as well as the existence of an in-kernel Just-In-Time (JIT) compiling machine for BPF were factors for the excellent performances of this tool.
Then in 2013, Alexei Starovoitov completely reshaped it, started to add new functionalities and to improve the performances of BPF. This new version is designated as eBPF (for “extended BPF”), while the former becomes cBPF (“classic” BPF). New features such as maps and tail calls appeared. The JIT machines were rewritten. The new language is even closer to native machine language than cBPF was. And also, new attach points in the kernel have been created.
Thanks to those new hooks, eBPF programs can be designed for a variety of use cases, that divide into two fields of applications. One of them is the domain of kernel tracing and event monitoring. BPF programs can be attached to kprobes and they compare with other tracing methods, with many advantages (and sometimes some drawbacks).
The other application domain remains network programming. In addition to socket filter, eBPF programs can be attached to tc (Linux traffic control tool) ingress or egress interfaces and perform a variety of packet processing tasks, in an efficient way. This opens new perspectives in the domain.
And eBPF performances are further leveraged through the technologies developed for the IO Visor project: new hooks have also been added for XDP (“eXpress Data Path”), a new fast path recently added to the kernel. XDP works in conjunction with the Linux stack, and relies on BPF to perform very fast packet processing.
Even some projects such as P4, Open vSwitch, consider or started to approach BPF. Some others, such as CETH, Cilium, are entirely based on it. BPF is buzzing, so we can expect a lot of tools and projects to orbit around it soon…
I've published a new HOWTO on my website:
Enrico already wrote about the Why (and the What, Who and When), so I'll just quote his conclusion and move on to the How.
I now have an XMPP setup which has all the features of the recent fancy chat systems, and on top of that it runs, client and server, on Free Software, which can be audited, it is federated and I can self-host my own server in my own VPS if I want to, with packages supported in Debian.Something I was used to and which came as standard on wheezy if you installed acpi-support was screen locking when you where suspending, hibernating, ...
This is something that I still haven't found on Jessie and which somebody had point me to solve via /lib/systemd/system-sleep/whatever hacking, but that didn't seem quite right, so I gave it a look again and this time I was able to add some config files at /etc/systemd and then a script which does what acpi-support used to do before
One of the most common questions I’m asked to cover when I discuss software architecture topics is the difference between the various application messaging protocols that exist today—issues like how and why the protocols came about, and which one should be used in a particular application.
This website contains latest news and background information regarding the SHA-1 freestart collision work from Marc Stevens (CWI, the Netherlands), Pierre Karpman (Inria, France and NTU Singapore) and Thomas Peyrin (NTU Singapore).
If you code a little, you know how handy an Integrated Development Environment (IE) can be. Java, C, Python, they all become a lot more accessible when the IDE software is checking the syntax for you, compiling in the background, or importing the libraries you need. On the other hand, if you are on Linux, you might also know how handy Vim can be when it comes to text editing. So naturally, you would like to get all the features of an IDE from Vim.
Recent Posts
Stuff The Internet Says On Scalability For August 28th, 2015x
7 Strategies for 10x Transformative Change
Ask HighScalability: Choose an Async App Server or Multiple Blocking Servers?
Stuff The Internet Says On Scalability For August 21st, 2015
The Microsoft Take on Containers and Docker
Sponsored Post: Surge, Redis Labs, Jut.io, VoltDB, Datadog, MongoDB, SignalFx, InMemory.Net, Couchbase, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7
How Autodesk Implemented Scalable Eventing over Mesos
Stuff The Internet Says On Scalability For August 14th, 2015
Why My Water Droplet Is Better Than Your Hadoop Cluster
How Google Invented an Amazing Datacenter Network Only They Could CreateAll Time Favorites
More Favorites...
YouTube Architecture
Plenty Of Fish Architecture
Google Architecture
How Twitter Stores 250 Million Tweets A Day Using MySQL
Scaling Twitter: Making Twitter 10000 Percent Faster
Flickr Architecture
Amazon Architecture
How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale
Stack Overflow Architecture
Facebook
An Unorthodox Approach to Database Design : The Coming of the Shard
Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud
Are Cloud Based Memory Architectures the Next Big Thing?
Latency is Everywhere and it Costs You Sales - How to Crush it
How will memristors change everything?
DataSift Architecture: Realtime Datamining At 120,000 Tweets Per Second
Useful Scalability Blogs
Scaling Traffic: People Pod Pool of On Demand Self Driving Robotic Cars who Automatically Refuel from Cheap Solar
VoltDB Decapitates Six SQL Urban Myths And Delivers Internet Scale OLTP In The Process
The Canonical Cloud Architecture
Justin.Tv's Live Video Broadcasting Architecture
Why Are Facebook, Digg, And Twitter So Hard To Scale?
What The Heck Are You Actually Using NoSQL For?
Playfish's Social Gaming Architecture - 50 Million Monthly Users And Growing
The Updated Big List Of Articles On The Amazon Outage
More Favorites...advertise
Login
Register
All Time Favorites
Useful Products
Useful Papers
Useful Strategies
Useful Blogs
Useful Books
Useful Conferences
Book Store
High Scalability RSS
High Scalability Comments RSS« The Mother of All Database Normalization Debates on Coding Horror | Main | Can cloud computing smite down evil zombie botnet armies? »
ZooKeeper - A Reliable, Scalable Distributed Coordination System
Tuesday, July 15, 2008 at 12:45AM
ZooKeeper is a high available and reliable coordination system. Distributed applications use ZooKeeper to store and mediate updates key configuration information. ZooKeeper can be used for leader election, group membership, and configuration maintenance. In addition ZooKeeper can be used for event notification, locking, and as a priority queue mechanism. It's a sort of central nervous system for distributed systems where the role of the brain is played by the coordination service, axons are the network, processes are the monitored and controlled body parts, and events are the hormones and neurotransmitters used for messaging. Every complex distributed application needs a coordination and orchestration system of some sort, so the ZooKeeper folks at Yahoo decide to build a good one and open source it for everyone to use.
The ESP8266 is the answer to “I want something with Wifi.” Surprisingly, there are a number of engineers and hobbyists who have not heard of this chip or have heard of it but don’t really understand what it is. It’s basically the answer to everything IoT to so many engineering problems that have plagued the hobbyist and commercial world alike.