Supporting Bookmate, from start-up to worldwide service

The CTO of Bookmate, Vasily Kolesnikov, explains: “Around 2 million books live in the Bookmate ecosystem, but every single book consists of multiple files, because every single chapter is a small, separated text file. So, when you have millions of books and you split those millions of books into chapters, you can imagine how many files we have.”

To give you an idea, you can upload your own ebooks to Bookmate (EPUB or FB2) without any limits, which translates to billions of files. “All ebooks that are uploaded to Bookmate are stored on Worldstream servers.” Operating your infrastructure isn’t exactly easy when you have billions of small files, so Bookmate decided on an in-house developed software defined storage solution. Also, to succeed they needed very specific hardware, which started their search for a competent provider.

From start-up to worldwide subscription service

Bookmate was researching different hosters in 2011, in either the Netherlands or in Germany. “Because the majority of our customers are in Europe, we wanted our infrastructure to be as close as possible to our customer base. Our back-end developers were building the infrastructure for the entire service.” Basically, from scratch, because they only had two bare metal nodes at that point, and Bookmate started searching for a solution with a reliable hoster. “With reliable we mean responsible with fast responsive support and the ability to provide custom hardware because we had very specific needs.”

At that point Bookmate was still a start-up and the price mattered. So, back then Bookmate started researching based on that and the first idea was to increase our plan with the current provider. But realized quite fast that they were not very reliable. "There were constant hardware issues and network outages back in 2011.” And Bookmate started looking elsewhere.

“We tried another provider, they were fine”, asserts Vasily firmly. “Their networking infrastructure was reliable, and they used branded servers such as Dell and Hewlett-Packard. But they were pricey, and their sales team was slow. We could request a hardware via e-mail or via their ticketing system and they would respond within like a week, so we would just wait for a week until they responded. If you are migrating from one hosting provider to another one, you kind of expect that the sales team will be working rapidly.” These experiences led them to rethink their strategy and write down their technical and service preferences.

The search for the right provider

“After continuing our search for a provider, we found Worldstream, and the main thing was they started responding immediately by e-mail, within minutes. On top of that your sales manager also proposed to switch to videoconferencing.

We used videoconferencing at that point so we could chat even faster, and we were like, wow, that’s like a real time interaction with your service provider. But more importantly Worldstream offered custom hardware configurations, which was very uncommon.”

The main reason Bookmate chose Worldstream as their provider was that they were able to provide custom hardware. “Not every single hoster would provide you with the hardware you request. They only had fixed configurations and that's it. Worldstream simply said ‘whatever you need, we can do it for you.’ And we responded: well, that's interesting.”

Another big reason for Bookmate to switch was support. “Worldstream support is the fastest so far. I mean I cannot, for my entire career, remember any other service provider who had better support than Worldstream. It doesn't matter what time of day it is. For example, if we have a broken hardware drive or a memory bank, even if it is midnight from Saturday to Sunday, you respond within minutes, and replace the malfunctioning bank or drive. But more importantly, if we need to upgrade something, we usually have it within minutes. At other providers we would wait two days just for a response and then the response would be something along the lines of: ‘Can you clarify something?’

When we were migrating from your first data center to the second data center in Naaldwijk your sales team proposed us a temporary custom configuration within the same deal, so we could migrate all our services from the old data center to the new data center seamlessly.

What if Bookmate chose an alternative solution?

Within the Worldstream infrastructure Bookmate uses their own software defined storage, and it is where they developed their own custom cloud service. “We use GlusterFS and it worked for us since the beginning and at the moment I think it's the only solution that can handle billions of small files, terabytes or even petabytes of small files. Originally, we started working on bare metal nodes, and recently we built our own on-premises cloud based on those nodes. This is our own virtualization platform, and we don't think that we're going to be changing anything soon. It was just monolithic applications running on bare metal nodes 10 years ago, but now everything is in containers.”

The reason why Bookmate is not thinking about changing anything soon is “because we tried to calculate the cost, and it turns out that if we migrate our infrastructure from Worldstream to a public cloud provider, it will increase monthly payments by about seven times.” If you have literally billions of small files, those files will be billed based on their networking and computing usage.

How Bookmate grew with Worldstream from the start

The cooperation between Bookmate and Worldstream has been ongoing for more than 10 years. Both parties have seen and helped each other grow business wise. Currently, the sole focus for Bookmate is to grow horizontally in European and Latin America countries. This translates to a need for more data, more workload, and more traffic. When Worldstream introduced another data center in Frankfurt they decided to move all their systems to Worldstream data centers.

“In 2012 all our production systems were migrated to Worldstream, and we only kept a few servers at another hosting provider for backups, because we didn't want to put all the eggs in one basket. Now, since the opening of your Frankfurt location in 2021 both production and backup systems are at Worldstream in different data centers. That's how much we trust you.”