How to create a EuroLinux mirror?

A mirror is a duplicate of the original resource, an identical version of it. Mirrors are extremely important to ensure higher availability in case of heavy load, the need to create copies in your own IT resources or to reduce the risk of not being able to access the original data. In this article, we will show how to make a public mirror of a EuroLinux system.

A list of selected public EuroLinux mirrors can be found at mirrors.cdn.euro-linux.com.

Creating a mirror

To create a mirror, you need a server with a disk size of at least 400 GB and Enterprise Linux 8. The detailed configuration of such a server is not included in this guide.

Note: for the purpose of this tutorial, we assume that the server-mirror will have its volume for the mirror mounted in the /mnt/mirror-volume/ directory.

We install nginx and certbot on the public server:

sudo dnf install -y certbot nginx

Next, we copy the following content to the /etc/nginx/nginx.conf file:

user       nginx;  ## Default: nobody
worker_processes  5;  ## Default: 1
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

worker_rlimit_nofile 8192;

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  include    mime.types;

  server {
      # Load configuration files for the default server block.
      include /etc/nginx/default.d/*.conf;

      location / {
          alias /mnt/mirror-volume/;
          autoindex on;
      }

      error_page 404 /404.html;
          location = /40x.html {
      }

      error_page 500 502 503 504 /50x.html;
          location = /50x.html {
      }
  }

  default_type application/octet-stream;
  log_format   main '$remote_addr - $remote_user [$time_local]  $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
  access_log  /var/log/nginx/access.log  main;
  sendfile     on;
  tcp_nopush   on;
  server_names_hash_bucket_size 128; # this seems to be required for some vhosts
 include /etc/nginx/conf.d/*.conf; 
}

To the file /etc/nginx/conf.d/mirror.conf copy the following content:

server {
    listen 443;
    root /mnt/mirror-volume ;
    ssl on;

    ssl_certificate /etc/letsencrypt/live/mirror/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mirror/privkey.pem;

    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;
    autoindex on;
    autoindex_exact_size off;
    autoindex_localtime on;
}

and to the file /etc/nginx/conf.d/redirect_to_https.conf the following content:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    return 301 https://$host$request_uri;
}

We stop firewalld and generate a certificate with certbot:

sudo systemctl stop firewalld
sudo certbot certonly --noninteractive --agree-tos --standalone --email <YOUR EMAIL> -d <YOUR DOMAIN NAME>
sudo ln -sfn /etc/letsencrypt/live/<YOUR DOMAIN NAME> /etc/letsencrypt/live/mirror

We turn on nginx and firewalld:

sudo systemctl enable nginx --now
sudo systemctl enable firewalld --now
sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

Add the following to /usr/local/bin/mirror.sh:

#!/bin/bash

# preparations
mkdir -p /mnt/mirror-volume/dist/eurolinux/server/{6,7}/x86_64/fbi
mkdir -p /mnt/mirror-volume/dist/eurolinux/server/8/{x86_64,aarch64,i386,i686}/{AppStream,BaseOS,PowerTools,HighAvailability,ResilientStorage}
mkdir -p /mnt/mirror-volume/dist/eurolinux/server/9/{x86_64,aarch64,i386,i686}/{AppStream,BaseOS,CRB,HighAvailability,ResilientStorage}

# general
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/TIME /mnt/mirror-volume/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/isos/ /mnt/mirror-volume/isos/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/security/ /mnt/mirror-volume/security/

# EL6
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/6/x86_64/fbi/ /mnt/mirror-volume/dist/eurolinux/server/6/x86_64/fbi/

# EL7
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/7/x86_64/fbi/ /mnt/mirror-volume/dist/eurolinux/server/7/x86_64/fbi/

# EL8
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/x86_64/AppStream/os/ /mnt/mirror-volume/dist/eurolinux/server/8/x86_64/AppStream/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/x86_64/BaseOS/os/ /mnt/mirror-volume/dist/eurolinux/server/8/x86_64/BaseOS/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/x86_64/PowerTools/os/ /mnt/mirror-volume/dist/eurolinux/server/8/x86_64/PowerTools/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/x86_64/HighAvailability/os/ /mnt/mirror-volume/dist/eurolinux/server/8/x86_64/HighAvailability/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/x86_64/ResilientStorage/os/ /mnt/mirror-volume/dist/eurolinux/server/8/x86_64/ResilientStorage/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/aarch64/AppStream/os/ /mnt/mirror-volume/dist/eurolinux/server/8/aarch64/AppStream/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/aarch64/BaseOS/os/ /mnt/mirror-volume/dist/eurolinux/server/8/aarch64/BaseOS/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/aarch64/PowerTools/os/ /mnt/mirror-volume/dist/eurolinux/server/8/aarch64/PowerTools/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/aarch64/HighAvailability/os/ /mnt/mirror-volume/dist/eurolinux/server/8/aarch64/HighAvailability/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/8/aarch64/ResilientStorage/os/ /mnt/mirror-volume/dist/eurolinux/server/8/aarch64/ResilientStorage/os/

# EL9
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/x86_64/BaseOS/os/ /mnt/mirror-volume/dist/eurolinux/server/9/x86_64/BaseOS/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/x86_64/AppStream/os/ /mnt/mirror-volume/dist/eurolinux/server/9/x86_64/AppStream/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/x86_64/CRB/os/ /mnt/mirror-volume/dist/eurolinux/server/9/x86_64/CRB/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/x86_64/Desktop/os/ /mnt/mirror-volume/dist/eurolinux/server/9/x86_64/Desktop/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/x86_64/HighAvailability/os/ /mnt/mirror-volume/dist/eurolinux/server/9/x86_64/HighAvailability/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/x86_64/ResilientStorage/os/ /mnt/mirror-volume/dist/eurolinux/server/9/x86_64/ResilientStorage/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/aarch64/BaseOS/os/ /mnt/mirror-volume/dist/eurolinux/server/9/aarch64/BaseOS/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/aarch64/AppStream/os/ /mnt/mirror-volume/dist/eurolinux/server/9/aarch64/AppStream/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/aarch64/CRB/os/ /mnt/mirror-volume/dist/eurolinux/server/9/aarch64/CRB/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/aarch64/HighAvailability/os/ /mnt/mirror-volume/dist/eurolinux/server/9/aarch64/HighAvailability/os/
rsync -avSH -f 'R .~tmp~' --delete-delay --delay-updates rsync://fbi2.cdn.euro-linux.com/repo/dist/eurolinux/server/9/aarch64/ResilientStorage/os/ /mnt/mirror-volume/dist/eurolinux/server/9/aarch64/ResilientStorage/os/

Finally, we add the following entries to crontab:

0 */3 * * * /bin/bash /usr/local/bin/mirror.sh
@monthly certbot certonly --nginx --noninteractive --agree-tos --email <TWÓJ EMAIL> -d <TWOJA NAZWA DOMENY>

Pull Request

We copy the mirror repository using the fork operation and create a pull request that will add a YAML file describing the mirror being created to the mirrors.d directory. We can use existing files as an example. The mirror does not have to provide all the protocols that the main EuroLinux mirror provides, but HTTP or HTTPS is required.

The YAML file should look like the template below:

---
name: <MIRROR NAME>
address:
  http: <MIRROR HTTP ADDRESS>
  https: <MIRROR HTTPS ADDRESS>
geolocation:
  country: <COUNTRY OF THE MIRROR>
update_frequency: 3h
sponsor: <YOUR NAME>
sponsor_url: <YOUR URL>
email: <YOUR EMAIL>
...

Summary

As you can see, making a EuroLinux mirror is relatively simple. However, you should take into account aspects beyond this tutorial. We are talking, among other things, about buying and running a server on a given hosting and configuring it in advance before starting the mirroring process.