Backblaze S3

Storage is a huge part of MD3’s Infrastructure and backups are a giant concern for us, that’s why we’ve been using AWS S3 for a long period of time.

Backblaze S3
Backblaze S3 Reviews

Mediatree core business is media broadcast monitoring, which involves recording and process huge amounts of media files. At the moment the infrastructure has in production around 500 international TV and Radio channels, where we record all the native quality of the channels, and process them in order to extract metadata (EPG information, thumbnails, closed caption and speech to text) and make it available for our customers.

MD3 is responsible for the administration of the IT Infrastructure of our parent company Mediatree.

Mediatree stores up to 1 year of content of this data, which makes storage a huge part of our Infrastructure. At the time, we have around 500TB of data in our storage, which is heavily mixed between big large files (native .ts files with raw quality) and a huge amount of small files (around 300M of mainly .jpeg, .srt and .xml).

B2 is the cloud storage offering from the very reputable online backup company, Backblaze. One of the biggest selling points of B2 is that it is fully compatible with Amazon’s S3 storage interface yet only costs about 25% of the official AWS offering, add into this considerably lower bandwidth fees and this service is already looking like a winner!
The S3 Compatible APIs are new, yet the company is already a well known innovator within the world of Infrastructure-as-a-Service (IaaS)—Backblaze has a devoted following and more than an.
The second major step for us was in May 2020, where they announced that BackBlaze B2 was now compatible with S3 API. This fully allowed us to migrate our backups fully from AWS S3 into BackBlaze B2 service, allowing us to save up to 80% in our invoice. The biggest advantage of BackBlaze B2 is the pricing model.
In May 2020, Backblaze, a founding Bandwidth Alliance partner announced S3 compatible APIs for their B2 Cloud Storage service. As a refresher, the Bandwidth Alliance is a group of forward-thinking cloud and networking companies that are committed to discounting or waiving data transfer fees for shared customers.

The S3 Compatible API for Backblaze B2 Cloud Storage allows 1000’s of integrations to work with B2 natively. If you’re new to the S3 Compatible API, please see our Getting Started Guide. If you have any trouble using S3FS with B2, please let us know by emailing us at b2feedback@backblaze.com.

Storage solutions are typically designed to handle well one of each case (big files vs small files), which puts us in one extreme case where we need to handle both types of data with similar performance.

Due to these requirements on the storage part of our infrastructure, backups are an even greater concern for us. Our platform is on 24/7 with very high SLA’s, and even one downtime of a few minutes has an impact on the production because our customers expect us to always deliver the full content without any cut in the video.

Our approach to backups has been to follow the 3-2-1 Rule, where we try to always have 3 copies of the data, in 2 different formats and 1 offsite.

To address the offsite part, we have been using for a long period of time AWS S3. In this case, we have daily backups from our main datacenter into a few AWS S3 buckets. The backups are mainly the media files and exported disks of our VM’s that we run in our datacenter.

We are constantly searching for newer solutions to improve our services and if possible reduce our costs. In 2018 our team went to NAB show in Las Vegas to meet with a few customers and also search for new solutions and ways to improve our services.

One of the stands there was BackBlaze. We had an interesting discussion with a person from BackBlaze but at the time, they were only based in the USA and had no datacenters in Europe, so it made no sense for us to use their services.

We enjoyed the company approach to the storage problems they were trying to solve (cheaper yet reliable alternative to other cloud storage services like AWS S3) and we kept following them to see the progress they were making.
Free download game killer swordigo.

For us, in August 2019, they made a major step forward: to open the first BackBlaze European datacentre. Late 2019, we started testing BackBlaze B2 service, however, our tests showed us that there was a major inconvenience with it, it was not S3 API compatible, which prevented us from using it in some of the backup software that we had in our production.

The second major step for us was in May 2020, where they announced that BackBlaze B2 was now compatible with S3 API. This fully allowed us to migrate our backups fully from AWS S3 into BackBlaze B2 service, allowing us to save up to 80% in our invoice.

The biggest advantage of BackBlaze B2 is the pricing model. When using BackBlaze we can have direct savings of 5x in comparison to AWS S3 and another big advantage is that the data egress pricing is also cheaper.

How to Setup AWS CLI to access BackBlaze B2

Requirements:

Important note: The location of the account is defined in the registration. If you want your account to be located in Europe other than the US, you need to specify it at the registration phase.

Access to BackBlaze Console.
AWS CLI installed
- AWS CLI is available in the official package repository of Ubuntu 18.04 LTS, if it is not installed you can easily install it with “sudo apt update && sudo apt install awscli”
Once inside the Backblaze console, we will go ahead and create our first bucket.

Select and Create Buckets

Inside the BackBlaze console, navigate to to “B2 Cloud Storage” and select “Buckets:

After selecting buckets, click on “Create a Bucket”

Insert the name of the backup and choose if you want your files to be public or private

At this stage, we should have our first bucket created. The important part for the rest of the tutorial is the “Endpoint”. This is the endpoint that we will have to set later when using the AWS S3 API in the CLI.

On the left menu, we need to navigate to “App Keys” to create our application keys, that in BackBlaze B2 are the equivalent of the AWS IAM Security keys.

By default, there is a Master Application Key. We will create a specific key that will be used only to access our new bucket.
Click on “Add a New Application Key”

A few parameters that can be set

While adding a new key, there are a few parameters that we can set:

Name of Key – the name of the key
Allow Access to Bucket – restrict the key to a specific bucket
Type of Access – give permission to Write/Read on the bucket
Allow List All Bucket Names – Allow this key to list all buckets in the account
Filename prefix – Allow only access to files that have a specific prefix
Duration – Specify access to files within the set duration

In our case, we will restrict the access to the bucket we created previously and give it Read/Write permissions.

After pressing “Create New Key” we should have a Success message that shows us our newly created key. From this, we need to save the KeyID and applicationKey

KeyID = AWS Access Key
ApplicationKey = AWS Secret Access Key

Our configuration work in the BackBlaze console is complete. We will keep it open to check later the result of our first file upload.

Now onto the AWS CLI configuration. We will add a new profile that will be used to access the BackBlaze B2 bucket.

To do this, we need to add a profile into our aws cli configuration that is usually located in our home folder ~/.aws

Open the file ~/.aws/config
add the following in the bottom of the file:
[profile b2]
region = eu-central-003
output = son
We should have something like:

Now on our credentials file at ~/.aws/credentials we will add:
[b2]
aws_access_key_id = 003bc0ef44688310000000005
aws_secret_access_key = [applicationKey]

The applicationKey is the value that we saved previously when we created the Key in BackBlaze Console.
The configuration steps are completed and now we should be able to use the AWS CLI to access our BackBlaze B2 storage.

In order to use it, we need to specify the –endpoint URL and also the profile

Example commands:

List all buckets: aws s3 ls –endpoint https://s3.eu-central-003.backblazeb2.com –profile b2

Copy a file: aws s3 cp file1 s3://backblaze-s3-api –endpoint https://s3.eu-central-003.backblazeb2.com –profile b2

In BackBlaze Console we can navigate to “Browse Files” and select our bucket “backblaze-s3-api” and we should be able to see our newly uploaded file.

Using BackBlaze B2 service could be a good alternative for a cheaper cloud backup solution and because most backup software already has support for AWS S3 storage it should be possible to migrate from AWS S3 to BackBlaze B2 easily.

This is an article written by Gonçalo Dias, Software Engineer@md3.

Gonçalo has 7 years of experience in systems administration, with a special taste for Linux systems and high availability. He is the MD3’s Head of IT, and is responsible for the management of the entire infrastructure of the Mediatree group and for the continuous development of all solutions and technologies used.

Backblaze (https://www.backblaze.com/) is a popular cloud storage service based in the United States. The service consists of an online backup space where you can save any type of file, using a client software to be installed on the local computer, which takes care of uploading and therefore backing up the data.

Backblaze security and reliability features make it an optimal solution for offsite backup of personal and business data. Among these features, we must first highlight redundancy, i.e. the fact that each uploaded file is stored redundantly on multiple units, on multiple servers and in multiple locations in the data center. Backblaze data centers are located in the United States (3) and Europe (1).

To further expand its offer and to also meet third-party software, Backblaze also supports the S3 protocol, effectively placing itself in direct competition with Amazon S3 Storage, with which it has always had a rivalry in terms of cost (lower) and popularity. Compatibility with the S3 protocol makes Backblaze perfectly usable with Iperius Backup, which supports any S3 compatible storage. We remember that Iperius also supports Wasabi, Aruba Cloud, Amazon S3, Minio, etc … Iperius also offers its S3 storage service (Iperius Storage S3) on TIER III and TIER IV datacenters, with advanced features, which can be purchased and activated from official website: https://www.iperiusstorage.com

Iperius is an excellent choice for backing up to Backblaze, because it adds enterprise-class features to simple file backup. Iperius can in fact perform drive image backups, backup of VMware ESXi and Hyper-V virtual machines, backup of databases such as SQL Server and MySQL, backup of Microsoft Exchange mail servers and Office 365. Iperius Backup can therefore significantly increase the completeness and the effectiveness of cloud backup on Backblaze, especially for companies and server environments.

In this short tutorial you will see how easy it is to perform a backup of folders on a Backblaze storage space, using the S3 protocol.

The first thing to do is to create a test account on Backblaze, and configure the access keys to authenticate and upload. From the main page of the official site https://www.backblaze.com/, create a new 10 GB storage test space, as shown in the image below:

You will have to specify an e-mail address and a password, and subsequently, a mobile phone number in order to receive a confirmation code and activate the account. Once this is done, go to the configuration panel and then to the section where you can create S3 credentials:

Click on “Add a New Application Key” and then specify the required fields as shown in the image below, then click on “Create New Key”:

As soon as the credentials are created, it will be possible to copy them and use them within Iperius. The password is only shown at this time, then it will no longer be visible. It will therefore be necessary to save it in a safe place, otherwise it will not be possible to recover it (if necessary, new credentials will have to be generated). The data to take note of are the keyID, the applicationKey and the S3 Endpoint address.

Now that our Backblaze account has been created and configured, let’s open Iperius and create a new backup job. Then add the folders you want to back up in the list of items:

In the next panel, “Destinations”, you will add a cloud destination and the related S3 account. In the image below you can see all the steps to add an S3 account with the Backblaze parameters:

Once the account has been added and saved, you can proceed to set the various destination options, such as the bucket name, the type of backup (full, incremental or differential), zip compression, etc.

Then save the destination by clicking on OK. You will then have your Backblaze destination in the list of destinations:

You can then proceed with the subsequent settings of the backup job, i.e. the schedule (to perform the backup in automatic mode based on days and times), the advanced settings, the email notifications and the execution of other processes. Once the configuration is complete, you can save the backup operation by giving it a name and clicking on “OK”:

You can then back up immediately, by right clicking on the just created backup job:

The tutorial has ended. We have seen how easy it is to perform an automatic backup on Backblaze, with the aim of having a secure copy of the data on a professional and always accessible offsite storage. Recall that, in addition to the basic functionality of Backblaze, Iperius offers very powerful features for backing up disk images, VMware ESXi and Hyper-V virtual machines, Exchange mail servers and Office 365. Iperius Backup and Backblaze S3 are an extremely powerful and complete combination to protect the IT infrastructure of any company, with absolutely competitive costs.

Backblaze S3

Finally, let’s not forget that Iperius in turn offers its Storage S3 service on advanced TIER IV datacenters, which allows you to perform the same operations that you can do with Backblaze S3, but offering a unified interface for license management and a web console for viewing and downloading files. The service is Iperius Storage S3, and can be purchased and activated from the official website: https://www.iperiusstorage.com

Backblaze S3 Reviews

(Italian, French, German, Spanish, Portuguese (Brazil))