Home  > Resources  > Blog

GitLab Administration

 
May 2, 2023 by Bibhas Bhattacharya
Category: DevOps

This tutorial is adapted from the Web Age course https://www.webagesolutions.com/courses/WA3255-introduction-to-gitlab-administration.

1.1 GitLab Administrator

  • Using GitLab.com
    • Only GitLab team members have access to administration tools and settings.
  • Using a self-managed GitLab instance
    • You must learn how to administer.
  • Only administrator users can access GitLab administration tools and settings.
  • An in-depth guide to full administration efforts is available here:

https://docs.gitlab.com/ee/administration/

1.2 GitLab Distributions

  • GitLab has two product distributions available through different subscriptions:
    • The open-source GitLab Community Edition (CE).
    • The open-core GitLab Enterprise Edition (EE).
  • You can install either GitLab CE or GitLab EE.
    • However, the features you have access to depend on your chosen subscription.
  • GitLab CE installations have access only to Free features.
  • You can install GitLab on most GNU/Linux distributions, on several cloud providers, and in Kubernetes clusters.
  • To get the best experience, you should balance performance, reliability, and ease of administration (backups, upgrades, and troubleshooting) with the cost of hosting.

1.3 Installing GitLab

Standalone installation

  • Linux package – Official deb/rpm packages (known as Omnibus GitLab).
    • GitLab package and dependent components, including PostgreSQL, Redis, and Sidekiq. This version is also used on GitLab.com site. https://docs.gitlab.com/omnibus/
  • Helm chart – A chart for installing a cloud-native version of GitLab and its components on Kubernetes.
  • Docker – The GitLab packages in a Docker container.
  • Source – GitLab and its components from scratch.
  • GitLab Environment Toolkit (GET) – A set of automation tools.
    • Use to deploy a reference architecture on most major cloud providers. Has some limitations and manual setup for production environments.
  • GitLab Operator – An installation and management method that follows the Kubernetes Operator pattern.
    • Use to run GitLab in an OpenShift environment.

Cloud Provider Installations

Amazon AWS (HA)

1.4 Installing GitLab on Linux

  • Linux installation is quicker to install, easier to upgrade, and contains features to enhance reliability not found in other methods.

1.5 Guide to Linux Installations

  • Install and configure the necessary dependencies
    • sudo apt-get update
    • sudo apt-get install -y curl openssh-server ca-certificates tzdata perl
  • Next, install Postfix (or Sendmail) to send notification emails.
    • sudo apt-get install -y postfix
  • Add the GitLab package repository and install the package
    • curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash
  • Next, install the GitLab package.
    • Ensure your DNS is set up properly, and change the URL listed https://gitlab.example.com to access your GitLab instance.
    • The installation will automatically configure and start GitLab at that URL.
    • If a password is not specified, a random password will be automatically generated.
    • sudo EXTERNAL_URL=”https://gitlab.example.com” apt-get install gitlab-ee
  • Browse to the hostname and login
    • The random password will be stored at /etc/gitlab/initial_root_password for 24 hours. Use this password with username root to login.

1.6 GitLab on Amazon AWS

  • In addition to having a basic familiarity with AWS and Amazon EC2, you need:
    • An AWS account
    • To create or upload an SSH key to connect to the instance via SSH
    • A domain name for the GitLab instance
    • An SSL/TLS certificate to secure your domain.
      • If you do not already own one, you can provision a free public SSL/TLS certificate through AWS Certificate Manager

Recommended Architecture on AWS

1.7 GitLab on Amazon Documentation

1.8 Official GitLab releases as AMIs

  • GitLab produces Amazon Machine Images (AMI) during the regular release process
  • The AMIs can be used for single instance GitLab installation or, by configuring /etc/gitlab/gitlab.rb, can be specialized for specific GitLab service roles (for example a Gitaly server).
    • Older releases remain available and can be used to migrate an older
  • GitLab server to AWS.
  • Currently the Amazon AMI uses the Amazon prepared Ubuntu AMI (x86 and ARM are available) as its starting point.
  • Instances running on Community Edition (CE) require a migration to Enterprise Edition (EE) to subscribe to the GitLab Premium or Ultimate plan.

1.9 Pick the GitLab AWS Edition

  • Many AWS editions are available from maketplace, choose official ones:
    • GitLab Community Edition: The open source version of GitLab.
    • GitLab Premium Enterprise Edition: If you want to unlock the enterprise features, a license is needed.
    • GitLab Ultimate Marketplace (Prelicensed): 5 user license built into per minute billing.
  • To launch an EC2 Machine with one of the listed AMIs, check the box at the start of the relevant row
  • AMI IDs are unique per region
  • If you are trying to restore from an older version of GitLab while moving to AWS, find the Enterprise and Community Editions before GitLab 11.10.3.

Costing will vary depending on the chosen type and support

1.10 EKS cluster provisioning best practices

  • GitLab can be used to provision an EKS cluster into AWS, however, it necessarily focuses on a basic EKS configuration.
  • Using the AWS tools can help with advanced cluster configuration, automation, and maintenance.
  • Clusters are purpose-built for:
    • EKS Clusters for GitLab Runners
    • Application Deployment Clusters for GitLab review apps
    • Application Deployment Cluster for production applications
  • GitLab “Cloud Native Hybrid” is a hybrid of the cloud-native technology Kubernetes (EKS) and EC2.
    • While as much of the GitLab application as possible runs in Kubernetes or on AWS services (PaaS), the GitLab service Gitaly must still be run on EC2.
    • Gitaly is a layer designed to overcome the limitations of the Git binaries in a horizontally scaled architecture. More shortly.

1.11 Use eksctl

Using eksctl enables the following when building an EKS Cluster:

  • Cluster configuration options include:
    • Selection of operating system: Amazon Linux 2, Windows, Bottlerocket
    • Selection of Hardware Architecture: x86, ARM, GPU
    • Selection of Kubernetes version
  • It can deploy high value-add items to the cluster, including:
    • A bastion host to keep the cluster endpoint private and possibly perform performance testing.
    • Prometheus and Grafana for monitoring.
  • EKS Autoscaler for automatic K8s Node scaling.
  • 2 or 3 Availability Zones (AZ) spread for balance between High Availability (HA) and cost control.
  • Ability to specify spot compute

1.12 Gitaly

  • Gitaly is an embedded service for Git Repository Storage.
  • Gitaly and Gitaly Cluster have been engineered by GitLab to overcome fundamental challenges with horizontal scaling of the open-source Git binaries that must be used on the service side of GitLab.
  • As part of Gitaly cluster consistency, Praefect nodes must occasionally vote on what data copy is the most accurate.
    • This requires an uneven number of Praefect nodes to avoid stalemates.
  • Minimum of three nodes.
  • Complete performance metrics should be collected for Gitaly instances for the identification of bottlenecks, like disk IO, network IO, or memory.
  • Gitaly functions as the primary Git Repository Storage in GitLab.
    • Prepares and caches Git pack files which informs some of the performance recommendations below.
      • Gitaly storage is expected to be local (not NFS).
      • Must use supported Elastic Network Adapter (ENA) advanced networking to ensure that cluster replication latency is not due to instance-level network I/O bottlenecks.

1.13 Get Started Administering GitLab

Authentication

  • Enforce two-factor authentication (2FA) for all users.
    • Add a backup email. If you lose access to your account, the GitLab Support team can help you more quickly.
    • Save or print your recovery codes. If you can’t access your authentication device, you can use these recovery codes.
    • Add an SSH key to your profile. You can generate new recovery codes as needed with SSH.
    • Enable personal access tokens. When using 2FA, you can use these tokens to access the GitLab API

Projects and groups

  • Organize your environment by configuring your groups and projects.
    • Projects: Designate a home for your files and code or track and organize issues in a business category.
    • Groups: Organize a collection of users or projects. Use these groups to quickly assign people and projects.
    • Roles: Define user access and visibility for your projects and groups.
  • General order for initial configuration is:
    • Create a project.
    • Create a group.
    • Add members to the group.
    • Create a subgroup.
    • Add members to the subgroup.
    • Enable external authorization control.
    • Sync group memberships by using LDAP.
    • Manage user access with inherited permissions.
      • Use up to 20 levels of subgroups to organize both teams and projects.
  • Alternatively import projects
    • You may need to import projects from external sources like GitHub, Bitbucket, or another instance of GitLab.
    • Many external sources can be imported into GitLab.
  • Consider repository mirroring—an alternative to project migrations
    • https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html
  • Schedule your project exports with GitLab import/export API.
  • Example GitHub importation see:
  • In GitLab, create a project:
  • In GitLab, on the top bar, select Main menu > Projects > View all projects.
  • On the right of the page, select New project.
  • Select Run CI/CD for external repository.
  • Select GitHub.
  • Login to Github.
  • Add a personal access token, paste the token.
  • Select List Repositories.
  • Select Connect to select the repository

GitLab instance security

  • Security is an important part of the on-boarding process.
  • Securing your instance protects your work and your organization.
  • Securing your instance involves:
    • Use a long root password, stored in a vault.
    • Install trusted SSL certificate and establish a process for renewal and revocation.
    • Configure SSH key restrictions per your organization’s guidelines.
    • Disable new sign-ups.
    • Require email confirmation.
    • Set password length limit, configure SSO or SAML user management.
    • Limit email domains if allowing sign-up.
    • Require two-factor authentication (2FA).
    • Disable password authentication for Git over HTTPS.
    • Set up email notifications for unknown sign-ins.
    • Configure user and IP rate limits.
    • Limit webhooks local access.
    • Set rate limits for protected paths.
    • Sign up for Security Alerts from the Communication Preference Center

Monitor GitLab performance

Components of monitoring

  • Web servers:
    • Handles server requests and facilitates other back-end service transactions.
      • Monitor CPU, memory, and network IO traffic to track the health of this node.
  • Workhorse:
    • Alleviates web traffic congestion from the main server.
      • Monitor latency spikes to track the health of this node.
  • Sidekiq:
    • Handles background operations that allow GitLab to run smoothly.
      • Monitor for long, unprocessed task queues to track the health of this node.

Back up your GitLab data

  • GitLab provides backup methods to keep your data safe and recoverable.
  • Could use a self-managed backup system or a GitLab SaaS database.
    • Decide on a backup strategy.
    • Consider writing a cron job to make daily backups.
    • Separately backup the configuration files.
    • Decide what should be left out of the backup.
    • Decide where to upload the backups.
    • Limit backup lifetime.
    • Run a test backup and restore.
    • Set up a way to periodically verify the backups

1.14 Example of Backing up GitLab SaaS

  • Backups of GitLab databases and file systems are taken every 24 hours, and are kept for two weeks on a rolling schedule.
    • All backups are encrypted.
  • GitLab SaaS creates backups to ensure your data is secure
    • Can’t use these methods to export or back up your data yourself.
  • Issues are stored in the database, not stored in Git itself.
  • You can use the project export option in:
    • The UI.
    • The API.
  • Group export does not export the projects in it, but does export:
    • Epics
    • Milestones
    • Boards
    • Labels
    • Additional items

1.15 Alternative backup strategies

  • In some situations, the Rake task for backups may not be the most optimal solution.

Option 1: File system snapshot

  • If your GitLab server contains a lot of Git repository data, you may find the GitLab backup script to be too slow, especially to an offsite location.
  • Slowness typically starts at a Git repository data size of around 200 GB. Consider using file system snapshots as part of your backup strategy.
  • For example, consider a GitLab server with the following components:
    • Using Omnibus GitLab
    • Hosted on AWS with an EBS drive containing an ext4 file system mounted at /var/opt/gitlab.
  • The EC2 instance meets the requirements for an application data backup by taking an EBS snapshot.
    • The backup includes all repositories, uploads, and PostgreSQL data.
  • In general, if you’re running GitLab on a virtualized server, you can create VM snapshots of the entire GitLab server.
  • It is common for a VM snapshot to require you to power down the server.

Option 2: GitLab Geo

  • Geo provides local, read-only instances of your GitLab instances.
  • While GitLab Geo helps remote teams work more efficiently by using a local GitLab node, it can also be used as a disaster recovery solution.
  • Geo replicates your database, your Git repositories, and a few other assets. Learn more about replication limitations

1.16 Restore a Backup

  • You can restore a backup only to the exact same version and type (Community Edition/Enterprise Edition) of GitLab on which it was created.
    • Review the Omnibus backup and restore documentation.
    • Review the Helm Chart backup and restore documentation.

1.17 Summary

  • Your need to work with the proper GitLab Distribution. CE and EE are the most common.
  • GitLab support from cloud providers is well documented, follow the guidelines though to not end up with surprise costs or performance issues
  • Administrating GitLab is straight forward but there are nuances for each environment to be aware of.

Follow Us

Blog Categories

Agile
Ajax & Web 2.0
Angular
Architecture
AWS
Big Data
Case Studies
Cloud
Data Science and Business Analytics
DevOps
Eclipse
IT
Java
JBoss
jQuery
Microsoft
Mobile Computing
News
New/Updated Courses
Scala
SOA & Web Services
Webinar
WebLogic
WebSphere