Proxmox log ошибок

/etc/nginx/snippets/letsencrypt.conf

location ^~ /.well-known/acme-challenge/ {
  allow all;
  root /var/lib/letsencrypt/;
  default_type "text/plain";
  try_files $uri =404;
}

Конфигурация сайта в NGINX

upstream proxmox1.domain.name  {
      server 127.0.0.1:8006;
}

server {
    listen 80;
    server_name proxmox1.domain.name;

    include snippets/letsencrypt.conf;
    return 301 https://$host$request_uri;
}


server {
    listen 443 ssl;

    server_name  proxmox1.domain.name;

    access_log  /var/log/nginx/proxmox1.domain.name.access.log;
    error_log  /var/log/nginx/proxmox1.domain.name.error.log;

    include snippets/letsencrypt.conf;

    ssl_certificate /etc/letsencrypt/live/proxmox1.domain.name/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/proxmox1.domain.name/privkey.pem;

    location / {
        proxy_pass  https://proxmox1.domain.name;
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503                                                                              http_504;
        proxy_redirect off;
        proxy_buffering off;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
   
}

Настройка /dev/sda

parted /dev/sda

(parted) print
Model: ATA SAMSUNG MZ7LM480 (scsi)
Disk /dev/sda: 480GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  4296MB  4295MB  primary               raid
 2      4296MB  4833MB  537MB   primary               raid
 3      4833MB  37,0GB  32,2GB  primary               raid

(parted) mkpart
Partition type?  primary/extended? primary
File system type?  [ext2]? zfs
Start? 33GB
End? 480GB
Warning: You requested a partition from 33,0GB to 480GB (sectors 64453125..937500000).
The closest location we can manage is 37,0GB to 480GB (sectors 72353792..937703087).
Is this still acceptable to you?
Yes/No? yes

PROXMOX VE ADMINISTRATION GUIDE

RELEASE 5.2

August 8, 2018

Proxmox Server Solutions Gmbh

www.proxmox.com

Proxmox VE Administration Guide ii

Copyright ©2017 Proxmox Server Solutions Gmbh

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free

Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with

no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.

A copy of the license is included in the section entitled «GNU Free Documentation License».

Proxmox VE Administration Guide iii

Contents

1 Introduction 1

1.1 Central Management ………………………………… 2

1.2 Flexible Storage ………………………………….. 3

1.3 Integrated Backup and Restore …………………………… 3

1.4 High Availability Cluster ………………………………. 3

1.5 Flexible Networking …………………………………. 4

1.6 Integrated Firewall …………………………………. 4

1.7 Why Open Source …………………………………. 4

1.8 Your benefit with Proxmox VE ……………………………. 4

1.9 Getting Help ……………………………………. 5

1.9.1 Proxmox VE Wiki ………………………………. 5

1.9.2 Community Support Forum ………………………….. 5

1.9.3 Mailing Lists …………………………………. 5

1.9.4 Commercial Support …………………………….. 5

1.9.5 Bug Tracker …………………………………. 5

1.10 Project History …………………………………… 5

1.11 Improving the Proxmox VE Documentation ……………………… 6

2 Installing Proxmox VE 7

2.1 System Requirements ……………………………….. 7

2.1.1 Minimum Requirements, for Evaluation …………………….. 7

2.1.2 Recommended System Requirements …………………….. 8

2.1.3 Simple Performance Overview ………………………… 8

2.1.4 Supported web browsers for accessing the web interface . . . . . . . . . . . . . . . . 8

2.2 Using the Proxmox VE Installation CD-ROM ……………….…….. 8

2.2.1 Advanced LVM Configuration Options …………………….. 14

2.2.2 ZFS Performance Tips ……………………………. 15

2.3 Install Proxmox VE on Debian ……………………………. 15

Proxmox VE Administration Guide iv

2.4 Install from USB Stick ……………………………….. 16

2.4.1 Prepare a USB flash drive as install medium ………………….. 16

2.4.2 Instructions for GNU/Linux ………………………….. 16

2.4.3 Instructions for OSX …………………………….. 17

2.4.4 Instructions for Windows …………………………… 17

2.4.5 Boot your server from USB media ………………………. 17

3 Host System Administration 18

3.1 Package Repositories ……………………………….. 18

3.1.1 Proxmox VE Enterprise Repository ………..…………….. 19

3.1.2 Proxmox VE No-Subscription Repository ……………………. 19

3.1.3 Proxmox VE Test Repository …………….…………… 19

3.1.4 SecureApt ………………………………….. 20

3.2 System Software Updates ……………………………… 20

3.3 Network Configuration ……………………………….. 21

3.3.1 Naming Conventions …………………………….. 21

3.3.2 Choosing a network configuration …………………….... 22

3.3.3 Default Configuration using a Bridge ……………………… 22

3.3.4 Routed Configuration …………………………….. 23

3.3.5 Masquerading (NAT) with iptables . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.3.6 Linux Bond …………………………………. 24

3.3.7 VLAN 802.1Q …..……………………………. 26

3.4 Time Synchronization ……………………………….. 28

3.4.1 Using Custom NTP Servers …………………………. 28

3.5 External Metric Server ……………………………….. 29

3.5.1 Graphite server configuration …………………………. 29

3.5.2 Influxdb plugin configuration …………………….…… 29

3.6 Disk Health Monitoring ……………………………….. 30

3.7 Logical Volume Manager (LVM) …………………….…….. 30

3.7.1 Hardware ………………………………….. 31

3.7.2 Bootloader …………………………………. 31

3.7.3 Creating a Volume Group …………………………… 31

3.7.4 Creating an extra LV for /var/lib/vz . . . . . . . . . . . . . . . . . . . . . . . . 32

3.7.5 Resizing the thin pool ………………..…………… 32

3.7.6 Create a LVM-thin pool ……………………………. 33

3.8 ZFS on Linux ……………………………………. 33

Proxmox VE Administration Guide v

3.8.1 Hardware ………………………………….. 34

3.8.2 Installation as Root File System ……………………….. 34

3.8.3 Bootloader …………………………………. 35

3.8.4 ZFS Administration ……………………………… 35

3.8.5 Activate E-Mail Notification ………………………….. 37

3.8.6 Limit ZFS Memory Usage ………………………….. 38

3.9 Certificate Management ………………………………. 39

3.9.1 Certificates for communication within the cluster ………………… 39

3.9.2 Certificates for API and web GUI ……………………….. 39

4 Hyper-converged Infrastructure 43

4.1 Benefits of a Hyper-Converged Infrastructure (HCI) with Proxmox VE . . . . . . . . . . . . . . 43

4.2 Manage Ceph Services on Proxmox VE Nodes ……………………. 44

4.2.1 Precondition ………………………………… 45

4.2.2 Installation of Ceph Packages ………………………… 46

4.2.3 Creating initial Ceph configuration ………………………. 46

4.2.4 Creating Ceph Monitors …………………………… 47

4.2.5 Creating Ceph Manager …………………………… 47

4.2.6 Creating Ceph OSDs …………………………….. 48

4.2.7 Creating Ceph Pools …………………………….. 50

4.2.8 Ceph CRUSH & device classes ……………………….. 51

4.2.9 Ceph Client …………………………………. 53

5 Graphical User Interface 54

5.1 Features ……………………………………… 54

5.2 Login ……………………………………….. 55

5.3 GUI Overview …………………………………… 55

5.3.1 Header …………………………………… 56

5.3.2 Resource Tree ……………………………….. 57

5.3.3 Log Panel ………………………………….. 57

5.4 Content Panels …………………………………… 58

5.4.1 Datacenter …………………………………. 58

5.4.2 Nodes ……………………………………. 59

5.4.3 Guests …………………………………… 60

5.4.4 Storage …………………………………… 62

5.4.5 Pools ……………………………………. 63

Proxmox VE Administration Guide vi

6 Cluster Manager 64

6.1 Requirements …………………………………… 64

6.2 Preparing Nodes ..………………………………… 65

6.3 Create the Cluster …………………………………. 65

6.3.1 Multiple Clusters In Same Network ………………………. 65

6.4 Adding Nodes to the Cluster …………………………….. 66

6.4.1 Adding Nodes With Separated Cluster Network ………………… 67

6.5 Remove a Cluster Node ………………………………. 67

6.5.1 Separate A Node Without Reinstalling …………………….. 68

6.6 Quorum ………………………………………. 70

6.7 Cluster Network ………………………………….. 70

6.7.1 Network Requirements ……………………………. 70

6.7.2 Separate Cluster Network ………………………….. 71

6.7.3 Redundant Ring Protocol …………………………… 74

6.7.4 RRP On Cluster Creation …………………….…….. 74

6.7.5 RRP On Existing Clusters ………………………….. 74

6.8 Corosync Configuration ………………………………. 75

6.8.1 Edit corosync.conf ……………………………… 75

6.8.2 Troubleshooting ……………………………….. 76

6.8.3 Corosync Configuration Glossary ………………………. 77

6.9 Cluster Cold Start …………………………..…….. 77

6.10 Guest Migration ………………………………….. 78

6.10.1 Migration Type ……………………………….. 78

6.10.2 Migration Network ……………………………… 78

7 Proxmox Cluster File System (pmxcfs) 80

7.1 POSIX Compatibility ………………………………… 80

7.2 File Access Rights …………………………………. 81

7.3 Technology …………………………………….. 81

7.4 File System Layout …………………………………. 81

7.4.1 Files …………………………………….. 81

7.4.2 Symbolic links ………………………………… 82

7.4.3 Special status files for debugging (JSON) . . . . . . . . . . . . . . . . . . . . . . . . 82

7.4.4 Enable/Disable debugging ………………………….. 82

7.5 Recovery ……………………………………… 82

7.5.1 Remove Cluster configuration ………..………………. 82

7.5.2 Recovering/Moving Guests from Failed Nodes . . . . . . . . . . . . . . . . . . . . . . 83

Proxmox VE Administration Guide vii

8 Proxmox VE Storage 84

8.1 Storage Types …………………………………… 84

8.1.1 Thin Provisioning ………………………………. 85

8.2 Storage Configuration ……………………………….. 85

8.2.1 Storage Pools ………………………………… 86

8.2.2 Common Storage Properties …………………………. 86

8.3 Volumes ……………………………………… 87

8.3.1 Volume Ownership ……………………………… 88

8.4 Using the Command Line Interface …………………………. 88

8.4.1 Examples ………………………………….. 88

8.5 Directory Backend …………………………………. 89

8.5.1 Configuration ………………………………… 90

8.5.2 File naming conventions …………………………… 90

8.5.3 Storage Features ………………………………. 91

8.5.4 Examples ………………………………….. 91

8.6 NFS Backend ……………………………………. 92

8.6.1 Configuration ………………………………… 92

8.6.2 Storage Features ………………………………. 93

8.6.3 Examples ………………………………….. 93

8.7 CIFS Backend …………………………………… 93

8.7.1 Configuration ………………………………… 93

8.7.2 Storage Features ………………………………. 94

8.7.3 Examples ………………………………….. 95

8.8 GlusterFS Backend …………………………………. 95

8.8.1 Configuration ………………………………… 95

8.8.2 File naming conventions …………………………… 96

8.8.3 Storage Features ………………………………. 96

8.9 Local ZFS Pool Backend …………..………………….. 96

8.9.1 Configuration ………………………………… 96

8.9.2 File naming conventions …………………………… 97

8.9.3 Storage Features ………………………………. 97

8.9.4 Examples ………………………………….. 97

8.10 LVM Backend ……………………………………. 97

8.10.1 Configuration ………………………………… 98

8.10.2 File naming conventions …………………………… 98

Proxmox VE Administration Guide viii

8.10.3 Storage Features ………………………………. 98

8.10.4 Examples ………………………….………. 99

8.11 LVM thin Backend …………………………………. 99

8.11.1 Configuration ………………………………… 99

8.11.2 File naming conventions ……………………………100

8.11.3 Storage Features ……………………………….100

8.11.4 Examples ………………………….……….100

8.12 Open-iSCSI initiator …………………………………100

8.12.1 Configuration …………………………………100

8.12.2 File naming conventions ……………………………101

8.12.3 Storage Features ……………………………….101

8.12.4 Examples ………………………….……….101

8.13 User Mode iSCSI Backend ……………………..……….102

8.13.1 Configuration …………………………………102

8.13.2 Storage Features ……………………………….102

8.14 Ceph RADOS Block Devices (RBD) ………………………….102

8.14.1 Configuration …………………………………103

8.14.2 Authentication …………………………………104

8.14.3 Storage Features ……………………………….104

8.15 Ceph Filesystem (CephFS) ……………………………..104

8.15.1 Configuration …………………………………104

8.15.2 Authentication …………………………………105

8.15.3 Storage Features ……………………………….105

9 Storage Replication 107

9.1 Supported Storage Types ………………………………107

9.2 Schedule Format …………………………………..108

9.2.1 Detailed Specification ………………………….….108

9.2.2 Examples: …………………………………..109

9.3 Error Handling ……………………………………109

9.3.1 Possible issues ………………………………..109

9.3.2 Migrating a guest in case of Error ……………………….110

9.3.3 Example …………………………………..110

9.4 Managing Jobs ……………………………………111

9.5 Command Line Interface Examples ………………………….111

Proxmox VE Administration Guide ix

10 Qemu/KVM Virtual Machines 112

10.1 Emulated devices and paravirtualized devices ……………………..112

10.2 Virtual Machines Settings ………………………………113

10.2.1 General Settings ……………………………….113

10.2.2 OS Settings ………………………………….114

10.2.3 Hard Disk …………………………………..114

10.2.4 CPU ……………………………………..116

10.2.5 Memory ……………………………………119

10.2.6 Network Device ………………………………..121

10.2.7 USB Passthrough ……………………………….122

10.2.8 BIOS and UEFI ………………………………..123

10.2.9 Automatic Start and Shutdown of Virtual Machines ……………….123

10.3 Migration ………………………………………124

10.3.1 Online Migration ……………………………….124

10.3.2 Offline Migration ……………………………….125

10.4 Copies and Clones ………………………………….125

10.5 Virtual Machine Templates ………………………………126

10.6 Importing Virtual Machines and disk images ………………………127

10.6.1 Step-by-step example of a Windows OVF import ………..……….127

10.6.2 Adding an external disk image to a Virtual Machine ……………….128

10.7 Cloud-Init Support ………………………………….128

10.7.1 Preparing Cloud-Init Templates …………………………129

10.7.2 Deploying Cloud-Init Templates …………………………130

10.7.3 Cloud-Init specific Options …………………………..131

10.8 Managing Virtual Machines with qm ………………………….132

10.8.1 CLI Usage Examples ……………………………..132

10.9 Configuration …………………………………….132

10.9.1 File Format ………………………………….133

10.9.2 Snapshots ………………………………….133

10.9.3 Options ……………………………………134

10.10Locks ………….…………………………….154

Proxmox VE Administration Guide x

11 Proxmox Container Toolkit 155

11.1 Technology Overview …………………………………155

11.2 Security Considerations ……………………………….156

11.3 Guest Operating System Configuration ………………………..156

11.4 Container Images ………………………………….158

11.5 Container Storage ………………………………….159

11.5.1 FUSE Mounts …………………………………159

11.5.2 Using Quotas Inside Containers ………………………..159

11.5.3 Using ACLs Inside Containers …………………………160

11.5.4 Backup of Containers mount points ………………………160

11.5.5 Replication of Containers mount points …………………….160

11.6 Container Settings ………………………………….160

11.6.1 General Settings ……………………………….160

11.6.2 CPU ……………………………………..162

11.6.3 Memory ……………………………………163

11.6.4 Mount Points …………………………………163

11.6.5 Network ……………………………………166

11.6.6 Automatic Start and Shutdown of Containers . . . . . . . . . . . . . . . . . . . . . . 167

11.7 Backup and Restore …………………………………168

11.7.1 Container Backup ……………………………….168

11.7.2 Restoring Container Backups ………………………….168

11.8 Managing Containers with pct ……………………………169

11.8.1 CLI Usage Examples ……………………………..170

11.8.2 Obtaining Debugging Logs …………………………..170

11.9 Migration ………………………………………171

11.10Configuration ……………………………..……..171

11.10.1File Format ………………………………….171

11.10.2Snapshots ………………………………….172

11.10.3Options ……………………………………172

11.11Locks ………….…………………………….177

12 Proxmox VE Firewall 178

12.1 Zones ………………………………………..178

12.2 Configuration Files ………………………………….178

12.2.1 Cluster Wide Setup ………………………………179

12.2.2 Host Specific Configuration …………………………..180

Proxmox VE Administration Guide xi

12.2.3 VM/Container Configuration ………………………….181

12.3 Firewall Rules ……………………………………182

12.4 Security Groups …………………………………..184

12.5 IP Aliases ………………………………………184

12.5.1 Standard IP Alias local_network . . . . . . . . . . . . . . . . . . . . . . . . . . 184

12.6 IP Sets ……………………………………….185

12.6.1 Standard IP set management …………………………185

12.6.2 Standard IP set blacklist …………………………185

12.6.3 Standard IP set ipfilter-net*………………………185

12.7 Services and Commands ………………………….……186

12.8 Tips and Tricks ……………………………………186

12.8.1 How to allow FTP ……………………………….186

12.8.2 Suricata IPS integration …………………………….186

12.9 Notes on IPv6 ……………………………………187

12.10Ports used by Proxmox VE ………………………………187

13 User Management 188

13.1 Users ………………………………………..188

13.1.1 System administrator ……………………………..188

13.1.2 Groups ……………………………………189

13.2 Authentication Realms ………………………………..189

13.3 Two factor authentication ……………………………….190

13.4 Permission Management ……………………………….190

13.4.1 Roles …………………………………….191

13.4.2 Privileges …………………………………..191

13.4.3 Objects and Paths ………………………………192

13.4.4 Pools …………………………………….193

13.4.5 What permission do I need? ………………………….193

13.5 Command Line Tool …………………………………194

13.6 Real World Examples ………………………………..195

13.6.1 Administrator Group ……………………………..195

13.6.2 Auditors ……………………………………195

13.6.3 Delegate User Management ………………………….196

13.6.4 Pools …………………………………….196

Proxmox VE Administration Guide xii

14 High Availability 197

14.1 Requirements ……………………………………198

14.2 Resources ……………………………………..199

14.3 Management Tasks ………………………………….199

14.4 How It Works …………………………………….200

14.4.1 Service States ………………………………..201

14.4.2 Local Resource Manager ……………………………202

14.4.3 Cluster Resource Manager …………………………..203

14.5 Configuration …………………………………….203

14.5.1 Resources ………………………………….204

14.5.2 Groups ……………………………………206

14.6 Fencing ……………………………………….208

14.6.1 How Proxmox VE Fences ……………………………208

14.6.2 Configure Hardware Watchdog ……………………..….209

14.6.3 Recover Fenced Services …………………………..209

14.7 Start Failure Policy ………………………………….209

14.8 Error Recovery ……………………………………210

14.9 Package Updates …………………………………..210

14.10Node Maintenance .…………………………………210

14.10.1Shutdown ………………………..…………210

14.10.2Reboot ……………………………………211

14.10.3Manual Resource Movement ………………………….211

15 Backup and Restore 212

15.1 Backup modes ……………………………………212

15.2 Backup File Names …………………………….……214

15.3 Restore ……………………………………….214

15.3.1 Bandwidth Limit ………………………………..214

15.4 Configuration …………………………………….215

15.5 Hook Scripts …………………………………….216

15.6 File Exclusions ……………………………………216

15.7 Examples ………………………………………217

Proxmox VE Administration Guide xiii

16 Important Service Daemons 218

16.1 pvedaemon — Proxmox VE API Daemon ………………………..218

16.2 pveproxy — Proxmox VE API Proxy Daemon ………………………218

16.2.1 Host based Access Control …………………………..218

16.2.2 SSL Cipher Suite ……………………………….219

16.2.3 Diffie-Hellman Parameters …………………………..219

16.2.4 Alternative HTTPS certificate ………………………….219

16.3 pvestatd — Proxmox VE Status Daemon ………………………..219

16.4 spiceproxy — SPICE Proxy Service …………………………..219

16.4.1 Host based Access Control …………………………..220

17 Useful Command Line Tools 221

17.1 pvesubscription — Subscription Management ………………………221

17.2 pveperf — Proxmox VE Benchmark Script ……………………….221

17.3 Shell interface for the Proxmox VE API ………………………..222

17.3.1 EXAMPLES ………………………………….222

17.4 Proxmox Node Management ……………………………..222

17.4.1 EXAMPLES ………………………………….222

18 Frequently Asked Questions 223

19 Bibliography 226

19.1 Books about Proxmox VE ………………………………226

19.2 Books about related technology ……………………………226

19.3 Books about related topics ………………………………227

A Command Line Interface 228

A.1 Output format options [FORMAT_OPTIONS] …………………….228

A.2 pvesm — Proxmox VE Storage Manager ………………………..229

A.3 pvesubscription — Proxmox VE Subscription Manager …………………238

A.4 pveperf — Proxmox VE Benchmark Script ……………………….239

A.5 pveceph — Manage CEPH Services on Proxmox VE Nodes ……………….239

A.6 pvenode — Proxmox VE Node Management ………………………243

A.7 pvesh — Shell interface for the Proxmox VE API …………………….246

A.8 qm — Qemu/KVM Virtual Machine Manager ………………………248

A.9 qmrestore — Restore QemuServer vzdump Backups . . . . . . . . . . . . . . . . . . . . . . 272

A.10 pct — Proxmox Container Toolkit ……………………………273

Proxmox VE Administration Guide xiv

A.11 pveam — Proxmox VE Appliance Manager ……………………….288

A.12 pvecm — Proxmox VE Cluster Manager ………………………..289

A.13 pvesr — Proxmox VE Storage Replication ……………………….292

A.14 pveum — Proxmox VE User Manager …………………………296

A.15 vzdump — Backup Utility for VMs and Containers . . . . . . . . . . . . . . . . . . . . . . . . 300

A.16 ha-manager — Proxmox VE HA Manager ……………………….302

B Service Daemons 307

B.1 pve-firewall — Proxmox VE Firewall Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . 307

B.2 pvedaemon — Proxmox VE API Daemon ……………………….308

B.3 pveproxy — Proxmox VE API Proxy Daemon ………………………309

B.4 pvestatd — Proxmox VE Status Daemon ………………………..309

B.5 spiceproxy — SPICE Proxy Service ………………………….310

B.6 pmxcfs — Proxmox Cluster File System ………………………..311

B.7 pve-ha-crm — Cluster Resource Manager Daemon …………………..311

B.8 pve-ha-lrm — Local Resource Manager Daemon . . . . . . . . . . . . . . . . . . . . . . . . 312

C Configuration Files 313

C.1 Datacenter Configuration ……………………………….313

C.1.1 File Format ………………………………….313

C.1.2 Options ……………………………………313

D Firewall Macro Definitions 316

E GNU Free Documentation License 330

Proxmox VE Administration Guide 1 / 336

Chapter 1

Introduction

Proxmox VE is a platform to run virtual machines and containers. It is based on Debian Linux, and completely

open source. For maximum flexibility, we implemented two virtualization technologies — Kernel-based Virtual

Machine (KVM) and container-based virtualization (LXC).

One main design goal was to make administration as easy as possible. You can use Proxmox VE on a

single node, or assemble a cluster of many nodes. All management tasks can be done using our web-based

management interface, and even a novice user can setup and install Proxmox VE within minutes.

Proxmox VE Administration Guide 2 / 336

1.1 Central Management

While many people start with a single node, Proxmox VE can scale out to a large set of clustered nodes.

The cluster stack is fully integrated and ships with the default installation.

Unique Multi-Master Design

The integrated web-based management interface gives you a clean overview of all your KVM guests

and Linux containers and even of your whole cluster. You can easily manage your VMs and con-

tainers, storage or cluster from the GUI. There is no need to install a separate, complex, and pricey

management server.

Proxmox Cluster File System (pmxcfs)

Proxmox VE uses the unique Proxmox Cluster file system (pmxcfs), a database-driven file system for

storing configuration files. This enables you to store the configuration of thousands of virtual machines.

By using corosync, these files are replicated in real time on all cluster nodes. The file system stores

all data inside a persistent database on disk, nonetheless, a copy of the data resides in RAM which

provides a maximum storage size is 30MB — more than enough for thousands of VMs.

Proxmox VE is the only virtualization platform using this unique cluster file system.

Web-based Management Interface

Proxmox VE is simple to use. Management tasks can be done via the included web based manage-

ment interface — there is no need to install a separate management tool or any additional management

node with huge databases. The multi-master tool allows you to manage your whole cluster from any

node of your cluster. The central web-based management — based on the JavaScript Framework (Ex-

tJS) — empowers you to control all functionalities from the GUI and overview history and syslogs of each

single node. This includes running backup or restore jobs, live-migration or HA triggered activities.

Command Line

For advanced users who are used to the comfort of the Unix shell or Windows Powershell, Proxmox

VE provides a command line interface to manage all the components of your virtual environment. This

command line interface has intelligent tab completion and full documentation in the form of UNIX man

pages.

REST API

Proxmox VE uses a RESTful API. We choose JSON as primary data format, and the whole API is for-

mally defined using JSON Schema. This enables fast and easy integration for third party management

tools like custom hosting environments.

Role-based Administration

You can define granular access for all objects (like VMs, storages, nodes, etc.) by using the role based

user- and permission management. This allows you to define privileges and helps you to control

access to objects. This concept is also known as access control lists: Each permission specifies a

subject (a user or group) and a role (set of privileges) on a specific path.

Authentication Realms

Proxmox VE supports multiple authentication sources like Microsoft Active Directory, LDAP, Linux PAM

standard authentication or the built-in Proxmox VE authentication server.

Proxmox VE Administration Guide 3 / 336

1.2 Flexible Storage

The Proxmox VE storage model is very flexible. Virtual machine images can either be stored on one or

several local storages or on shared storage like NFS and on SAN. There are no limits, you may configure as

many storage definitions as you like. You can use all storage technologies available for Debian Linux.

One major benefit of storing VMs on shared storage is the ability to live-migrate running machines without

any downtime, as all nodes in the cluster have direct access to VM disk images.

We currently support the following Network storage types:

• LVM Group (network backing with iSCSI targets)

• iSCSI target

• NFS Share

• CIFS Share

• Ceph RBD

• Directly use iSCSI LUNs

• GlusterFS

Local storage types supported are:

• LVM Group (local backing devices like block devices, FC devices, DRBD, etc.)

• Directory (storage on existing filesystem)

• ZFS

1.3 Integrated Backup and Restore

The integrated backup tool (vzdump) creates consistent snapshots of running Containers and KVM guests.

It basically creates an archive of the VM or CT data which includes the VM/CT configuration files.

KVM live backup works for all storage types including VM images on NFS, CIFS, iSCSI LUN, Ceph RBD or

Sheepdog. The new backup format is optimized for storing VM backups fast and effective (sparse files, out

of order data, minimized I/O).

1.4 High Availability Cluster

A multi-node Proxmox VE HA Cluster enables the definition of highly available virtual servers. The Proxmox

VE HA Cluster is based on proven Linux HA technologies, providing stable and reliable HA services.

Proxmox VE Administration Guide 4 / 336

1.5 Flexible Networking

Proxmox VE uses a bridged networking model. All VMs can share one bridge as if virtual network cables

from each guest were all plugged into the same switch. For connecting VMs to the outside world, bridges

are attached to physical network cards assigned a TCP/IP configuration.

For further flexibility, VLANs (IEEE 802.1q) and network bonding/aggregation are possible. In this way it is

possible to build complex, flexible virtual networks for the Proxmox VE hosts, leveraging the full power of the

Linux network stack.

1.6 Integrated Firewall

The integrated firewall allows you to filter network packets on any VM or Container interface. Common sets

of firewall rules can be grouped into “security groups”.

1.7 Why Open Source

Proxmox VE uses a Linux kernel and is based on the Debian GNU/Linux Distribution. The source code of

Proxmox VE is released under the GNU Affero General Public License, version 3. This means that you are

free to inspect the source code at any time or contribute to the project yourself.

At Proxmox we are committed to use open source software whenever possible. Using open source software

guarantees full access to all functionalities — as well as high security and reliability. We think that everybody

should have the right to access the source code of a software to run it, build on it, or submit changes back

to the project. Everybody is encouraged to contribute while Proxmox ensures the product always meets

professional quality criteria.

Open source software also helps to keep your costs low and makes your core infrastructure independent

from a single vendor.

1.8 Your benefit with Proxmox VE

• Open source software

• No vendor lock-in

• Linux kernel

• Fast installation and easy-to-use

• Web-based management interface

• REST API

• Huge active community

• Low administration costs and simple deployment

Proxmox VE Administration Guide 5 / 336

1.9 Getting Help

1.9.1 Proxmox VE Wiki

The primary source of information is the Proxmox VE Wiki. It combines the reference documentation with

user contributed content.

1.9.2 Community Support Forum

Proxmox VE itself is fully open source, so we always encourage our users to discuss and share their knowl-

edge using the Proxmox VE Community Forum. The forum is fully moderated by the Proxmox support team,

and has a quite large user base around the whole world. Needless to say that such a large forum is a great

place to get information.

1.9.3 Mailing Lists

This is a fast way to communicate via email with the Proxmox VE community

• Mailing list for users: PVE User List

The primary communication channel for developers is:

• Mailing list for developer: PVE development discussion

1.9.4 Commercial Support

Proxmox Server Solutions Gmbh also offers commercial Proxmox VE Subscription Service Plans. System

Administrators with a standard subscription plan can access a dedicated support portal with guaranteed re-

sponse time, where Proxmox VE developers help them should an issue appear. Please contact the Proxmox

sales team for more information or volume discounts.

1.9.5 Bug Tracker

We also run a public bug tracker at https://bugzilla.proxmox.com. If you ever detect an issue, you can file a

bug report there. This makes it easy to track its status, and you will get notified as soon as the problem is

fixed.

1.10 Project History

The project started in 2007, followed by a first stable version in 2008. At the time we used OpenVZ for

containers, and KVM for virtual machines. The clustering features were limited, and the user interface was

simple (server generated web page).

But we quickly developed new features using the Corosync cluster stack, and the introduction of the new

Proxmox cluster file system (pmxcfs) was a big step forward, because it completely hides the cluster com-

plexity from the user. Managing a cluster of 16 nodes is as simple as managing a single node.

Proxmox VE Administration Guide 6 / 336

We also introduced a new REST API, with a complete declarative specification written in JSON-Schema.

This enabled other people to integrate Proxmox VE into their infrastructure, and made it easy to provide

additional services.

Also, the new REST API made it possible to replace the original user interface with a modern HTML5

application using JavaScript. We also replaced the old Java based VNC console code with noVNC. So

you only need a web browser to manage your VMs.

The support for various storage types is another big task. Notably, Proxmox VE was the first distribution to

ship ZFS on Linux by default in 2014. Another milestone was the ability to run and manage Ceph storage on

the hypervisor nodes. Such setups are extremely cost effective.

When we started we were among the first companies providing commercial support for KVM. The KVM

project itself continuously evolved, and is now a widely used hypervisor. New features arrive with each

release. We developed the KVM live backup feature, which makes it possible to create snapshot backups on

any storage type.

The most notable change with version 4.0 was the move from OpenVZ to LXC. Containers are now deeply

integrated, and they can use the same storage and network features as virtual machines.

1.11 Improving the Proxmox VE Documentation

Depending on which issue you want to improve, you can use a variety of communication mediums to reach

the developers.

If you notice an error in the current documentation, use the Proxmox bug tracker and propose an alternate

text/wording.

If you want to propose new content, it depends on what you want to document:

• if the content is specific to your setup, a wiki article is the best option. For instance if you want to document

specific options for guest systems, like which combination of Qemu drivers work best with a less popular

OS, this is a perfect fit for a wiki article.

• if you think the content is generic enough to be of interest for all users, then you should try to get it into the

reference documentation. The reference documentation is written in the easy to use asciidoc document

format. Editing the official documentation requires to clone the git repository at git://git.proxmox.

com/git/pve-docs.git and then follow the README.adoc document.

Improving the documentation is just as easy as editing a Wikipedia article and is an interesting foray in the

development of a large opensource project.

Note

If you are interested in working on the Proxmox VE codebase, the Developer Documentation wiki article

will show you where to start.

Proxmox VE Administration Guide 7 / 336

Chapter 2

Installing Proxmox VE

Proxmox VE is based on Debian and comes with an installation CD-ROM which includes a complete Debian

system («stretch» for version 5.x) as well as all necessary Proxmox VE packages.

The installer just asks you a few questions, then partitions the local disk(s), installs all required packages,

and configures the system including a basic network setup. You can get a fully functional system within a

few minutes. This is the preferred and recommended installation method.

Alternatively, Proxmox VE can be installed on top of an existing Debian system. This option is only recom-

mended for advanced users since detail knowledge about Proxmox VE is necessary.

2.1 System Requirements

For production servers, high quality server equipment is needed. Keep in mind, if you run 10 Virtual Servers

on one machine and you then experience a hardware failure, 10 services are lost. Proxmox VE supports

clustering, this means that multiple Proxmox VE installations can be centrally managed thanks to the included

cluster functionality.

Proxmox VE can use local storage (DAS), SAN, NAS and also distributed storage (Ceph RBD). For details

see chapter storage Chapter 8.

2.1.1 Minimum Requirements, for Evaluation

• CPU: 64bit (Intel EMT64 or AMD64)

• Intel VT/AMD-V capable CPU/Mainboard for KVM Full Virtualization support

• RAM: 1 GB RAM, plus additional RAM used for guests

• Hard drive

• One NIC

Proxmox VE Administration Guide 8 / 336

2.1.2 Recommended System Requirements

• CPU: 64bit (Intel EMT64 or AMD64), Multi core CPU recommended

• Intel VT/AMD-V capable CPU/Mainboard for KVM Full Virtualization support

• RAM: 8 GB RAM, plus additional RAM used for guests

• Hardware RAID with batteries protected write cache (“BBU”) or flash based protection

• Fast hard drives, best results with 15k rpm SAS, Raid10

• At least two NICs, depending on the used storage technology you need more

2.1.3 Simple Performance Overview

On an installed Proxmox VE system, you can run the included pveperf script to obtain an overview of the

CPU and hard disk performance.

Note

this is just a very quick and general benchmark. More detailed tests are recommended, especially regard-

ing the I/O performance of your system.

2.1.4 Supported web browsers for accessing the web interface

To use the web interface you need a modern browser, this includes:

• Firefox, a release from the current year, or the latest Extended Support Release

• Chrome, a release from the current year

• the Microsoft currently supported versions of Internet Explorer (as of 2016, this means IE 11 or IE Edge)

• the Apple currently supported versions of Safari (as of 2016, this means Safari 9)

If Proxmox VE detects you’re connecting from a mobile device, you will be redirected to a lightweight touch-

based UI.

2.2 Using the Proxmox VE Installation CD-ROM

You can download the ISO from http://www.proxmox.com. It includes the following:

• Complete operating system (Debian Linux, 64-bit)

• The Proxmox VE installer, which partitions the hard drive(s) with ext4, ext3, xfs or ZFS and installs the

operating system.

• Proxmox VE kernel (Linux) with LXC and KVM support

Proxmox VE Administration Guide 9 / 336

• Complete toolset for administering virtual machines, containers and all necessary resources

• Web based management interface for using the toolset

Note

By default, the complete server is used and all existing data is removed.

Please insert the installation CD-ROM, then boot from that drive. Immediately afterwards you can choose

the following menu options:

Install Proxmox VE

Start normal installation.

Install Proxmox VE (Debug mode)

Start installation in debug mode. It opens a shell console at several installation steps, so that you

can debug things if something goes wrong. Please press CTRL-D to exit those debug consoles and

continue installation. This option is mostly for developers and not meant for general use.

Rescue Boot

This option allows you to boot an existing installation. It searches all attached hard disks and, if it finds

an existing installation, boots directly into that disk using the existing Linux kernel. This can be useful

if there are problems with the boot block (grub), or the BIOS is unable to read the boot block from the

disk.

Test Memory

Runs memtest86+. This is useful to check if your memory is functional and error free.

Proxmox VE Administration Guide 10 / 336

You normally select Install Proxmox VE to start the installation. After that you get prompted to select the

target hard disk(s). The Options button lets you select the target file system, which defaults to ext4. The

installer uses LVM if you select ext3,ext4 or xfs as file system, and offers additional option to restrict

LVM space (see below)

If you have more than one disk, you can also use ZFS as file system. ZFS supports several software RAID

levels, so this is specially useful if you do not have a hardware RAID controller. The Options button lets

you select the ZFS RAID level, and you can choose disks there.

Proxmox VE Administration Guide 11 / 336

The next page just ask for basic configuration options like your location, the time zone and keyboard layout.

The location is used to select a download server near you to speedup updates. The installer is usually able

to auto detect those setting, so you only need to change them in rare situations when auto detection fails, or

when you want to use some special keyboard layout not commonly used in your country.

Proxmox VE Administration Guide 12 / 336

You then need to specify an email address and the superuser (root) password. The password must have at

least 5 characters, but we highly recommend to use stronger passwords — here are some guidelines:

• Use a minimum password length of 12 to 14 characters.

• Include lowercase and uppercase alphabetic characters, numbers and symbols.

• Avoid character repetition, keyboard patterns, dictionary words, letter or number sequences, usernames,

relative or pet names, romantic links (current or past) and biographical information (e.g., ID numbers,

ancestors’ names or dates).

It is sometimes necessary to send notification to the system administrator, for example:

• Information about available package updates.

• Error messages from periodic CRON jobs.

All those notification mails will be sent to the specified email address.

Proxmox VE Administration Guide 13 / 336

The last step is the network configuration. Please note that you can use either IPv4 or IPv6 here, but not

both. If you want to configure a dual stack node, you can easily do that after installation.

Proxmox VE Administration Guide 14 / 336

If you press Next now, installation starts to format disks, and copies packages to the target. Please wait

until that is finished, then reboot the server.

Further configuration is done via the Proxmox web interface. Just point your browser to the IP address given

during installation (https://youripaddress:8006).

Note

Default login is «root» (realm PAM) and the root password is defined during the installation process.

2.2.1 Advanced LVM Configuration Options

The installer creates a Volume Group (VG) called pve, and additional Logical Volumes (LVs) called root,

data and swap. The size of those volumes can be controlled with:

hdsize

Defines the total HD size to be used. This way you can save free space on the HD for further partition-

ing (i.e. for an additional PV and VG on the same hard disk that can be used for LVM storage).

swapsize

Defines the size of the swap volume. The default is the size of the installed memory, minimum 4 GB

and maximum 8 GB. The resulting value cannot be greater than hdsize/8.

Proxmox VE Administration Guide 15 / 336

Note

If set to 0, no swap volume will be created.

maxroot

Defines the maximum size of the root volume, which stores the operation system. The maximum

limit of the root volume size is hdsize/4.

maxvz

Defines the maximum size of the data volume. The actual size of the data volume is:

datasize = hdsize — rootsize — swapsize — minfree

Where datasize cannot be bigger than maxvz.

Note

In case of LVM thin, the data pool will only be created if datasize is bigger than 4GB.

Note

If set to 0, no data volume will be created and the storage configuration will be adapted accordingly.

minfree

Defines the amount of free space left in LVM volume group pve. With more than 128GB storage

available the default is 16GB, else hdsize/8 will be used.

Note

LVM requires free space in the VG for snapshot creation (not required for lvmthin snapshots).

2.2.2 ZFS Performance Tips

ZFS uses a lot of memory, so it is best to add additional RAM if you want to use ZFS. A good calculation is

4GB plus 1GB RAM for each TB RAW disk space.

ZFS also provides the feature to use a fast SSD drive as write cache. The write cache is called the ZFS

Intent Log (ZIL). You can add that after installation using the following command:

zpool add <pool-name> log </dev/path_to_fast_ssd>

2.3 Install Proxmox VE on Debian

Proxmox VE ships as a set of Debian packages, so you can install it on top of a normal Debian installation.

After configuring the repositories, you need to run:

apt-get update

apt-get install proxmox-ve

Proxmox VE Administration Guide 16 / 336

Installing on top of an existing Debian installation looks easy, but it presumes that you have correctly installed

the base system, and you know how you want to configure and use the local storage. Network configuration

is also completely up to you.

In general, this is not trivial, especially when you use LVM or ZFS.

You can find a detailed step by step howto on the wiki.

2.4 Install from USB Stick

The Proxmox VE installation media is now a hybrid ISO image, working in two ways:

• An ISO image file ready to burn on CD

• A raw sector (IMG) image file ready to directly copy to flash media (USB Stick)

Using USB sticks is faster and more environmental friendly and therefore the recommended way to install

Proxmox VE.

2.4.1 Prepare a USB flash drive as install medium

In order to boot the installation media, copy the ISO image to a USB media.

First download the ISO image from https://www.proxmox.com/en/downloads/category/iso-images-pve

You need at least a 1 GB USB media.

Note

Using UNetbootin or Rufus does not work.

Important

Make sure that the USB media is not mounted and does not contain any important data.

2.4.2 Instructions for GNU/Linux

You can simply use dd on UNIX like systems. First download the ISO image, then plug in the USB stick. You

need to find out what device name gets assigned to the USB stick (see below). Then run:

dd if=proxmox-ve_*.iso of=/dev/XYZ bs=1M

Note

Be sure to replace /dev/XYZ with the correct device name.

Caution

Be very careful, and do not overwrite the hard disk!

Proxmox VE Administration Guide 17 / 336

Find Correct USB Device Name

You can compare the last lines of dmesg command before and after the insertion, or use the lsblk command.

Open a terminal and run:

lsblk

Then plug in your USB media and run the command again:

lsblk

A new device will appear, and this is the USB device you want to use.

2.4.3 Instructions for OSX

Open the terminal (query Terminal in Spotlight).

Convert the .iso file to .img using the convert option of hdiutil for example.

hdiutil convert -format UDRW -o proxmox-ve_*.dmg proxmox-ve_*.iso

Tip

OS X tends to put the .dmg ending on the output file automatically.

To get the current list of devices run the command again:

diskutil list

Now insert your USB flash media and run this command again to determine the device node assigned to

your flash media (e.g. /dev/diskX).

diskutil list

diskutil unmountDisk /dev/diskX

Note

replace X with the disk number from the last command.

sudo dd if=proxmox-ve_*.dmg of=/dev/rdiskN bs=1m

2.4.4 Instructions for Windows

Download Etcher from https://etcher.io , select the ISO and your USB Drive.

If this doesn’t work, alternatively use the OSForensics USB installer from http://www.osforensics.com/portability.html

2.4.5 Boot your server from USB media

Connect your USB media to your server and make sure that the server boots from USB (see server BIOS).

Then follow the installation wizard.

Proxmox VE Administration Guide 18 / 336

Chapter 3

Host System Administration

Proxmox VE is based on the famous Debian Linux distribution. That means that you have access to the

whole world of Debian packages, and the base system is well documented. The Debian Administrator’s

Handbook is available online, and provides a comprehensive introduction to the Debian operating system

(see [Hertzog13]).

A standard Proxmox VE installation uses the default repositories from Debian, so you get bug fixes and

security updates through that channel. In addition, we provide our own package repository to roll out all

Proxmox VE related packages. This includes updates to some Debian packages when necessary.

We also deliver a specially optimized Linux kernel, where we enable all required virtualization and container

features. That kernel includes drivers for ZFS, and several hardware drivers. For example, we ship Intel

network card drivers to support their newest hardware.

The following sections will concentrate on virtualization related topics. They either explains things which are

different on Proxmox VE, or tasks which are commonly used on Proxmox VE. For other topics, please refer

to the standard Debian documentation.

3.1 Package Repositories

All Debian based systems use APT as package management tool. The list of repositories is defined in /

etc/apt/sources.list and .list files found inside /etc/apt/sources.d/. Updates can be

installed directly using apt-get, or via the GUI.

Apt sources.list files list one package repository per line, with the most preferred source listed first.

Empty lines are ignored, and a #character anywhere on a line marks the remainder of that line as a com-

ment. The information available from the configured sources is acquired by apt-get update.

File /etc/apt/sources.list

deb http://ftp.debian.org/debian stretch main contrib

# security updates

deb http://security.debian.org stretch/updates main contrib

In addition, Proxmox VE provides three different package repositories.

Proxmox VE Administration Guide 19 / 336

3.1.1 Proxmox VE Enterprise Repository

This is the default, stable and recommended repository, available for all Proxmox VE subscription users. It

contains the most stable packages, and is suitable for production use. The pve-enterprise repository

is enabled by default:

File /etc/apt/sources.list.d/pve-enterprise.list

deb https://enterprise.proxmox.com/debian/pve stretch pve-enterprise

As soon as updates are available, the root@pam user is notified via email about the available new pack-

ages. On the GUI, the change-log of each package can be viewed (if available), showing all details of the

update. So you will never miss important security fixes.

Please note that and you need a valid subscription key to access this repository. We offer different support

levels, and you can find further details at http://www.proxmox.com/en/proxmox-ve/pricing.

Note

You can disable this repository by commenting out the above line using a #(at the start of the line).

This prevents error messages if you do not have a subscription key. Please configure the pve-no-

subscription repository in that case.

3.1.2 Proxmox VE No-Subscription Repository

As the name suggests, you do not need a subscription key to access this repository. It can be used for

testing and non-production use. Its not recommended to run on production servers, as these packages are

not always heavily tested and validated.

We recommend to configure this repository in /etc/apt/sources.list.

File /etc/apt/sources.list

deb http://ftp.debian.org/debian stretch main contrib

# PVE pve-no-subscription repository provided by proxmox.com,

# NOT recommended for production use

deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

# security updates

deb http://security.debian.org stretch/updates main contrib

3.1.3 Proxmox VE Test Repository

Finally, there is a repository called pvetest. This one contains the latest packages and is heavily used by

developers to test new features. As usual, you can configure this using /etc/apt/sources.list by

adding the following line:

Proxmox VE Administration Guide 20 / 336

sources.list entry for pvetest

deb http://download.proxmox.com/debian/pve stretch pvetest

Warning

the pvetest repository should (as the name implies) only be used for testing new features or bug

fixes.

3.1.4 SecureApt

We use GnuPG to sign the Release files inside those repositories, and APT uses that signatures to verify

that all packages are from a trusted source.

The key used for verification is already installed if you install from our installation CD. If you install by other

means, you can manually download the key with:

# wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg ←—

-O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg

Please verify the checksum afterwards:

# sha512sum /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg

ffb95f0f4be68d2e753c8875ea2f8465864a58431d5361e88789568673551501ae574283a4e0492f17d79dc67edfb173a56a6304dea39e01f249ebdabc9f074a ←—

/etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg

or

# md5sum /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg

511d36d0f1350c01c42a3dc9f3c27939 /etc/apt/trusted.gpg.d/proxmox-ve-release ←—

-5.x.gpg

3.2 System Software Updates

We provide regular package updates on all repositories. You can install those update using the GUI, or you

can directly run the CLI command apt-get:

apt-get update

apt-get dist-upgrade

Note

The apt package management system is extremely flexible and provides countless of feature — see man

apt-get or [Hertzog13] for additional information.

You should do such updates at regular intervals, or when we release versions with security related fixes.

Major system upgrades are announced at the Proxmox VE Community Forum. Those announcement also

contain detailed upgrade instructions.

Tip

We recommend to run regular upgrades, because it is important to get the latest security updates.

Proxmox VE Administration Guide 21 / 336

3.3 Network Configuration

Network configuration can be done either via the GUI, or by manually editing the file /etc/network/

interfaces, which contains the whole network configuration. The interfaces(5) manual page

contains the complete format description. All Proxmox VE tools try hard to keep direct user modifications,

but using the GUI is still preferable, because it protects you from errors.

Once the network is configured, you can use the Debian traditional tools ifup and ifdown commands to

bring interfaces up and down.

Note

Proxmox VE does not write changes directly to /etc/network/interfaces. Instead, we write into

a temporary file called /etc/network/interfaces.new, and commit those changes when you

reboot the node.

3.3.1 Naming Conventions

We currently use the following naming conventions for device names:

• Ethernet devices: en*, systemd network interface names. This naming scheme is used for new Proxmox

VE installations since version 5.0.

• Ethernet devices: eth[N], where 0 ≤N (eth0,eth1, . . . ) This naming scheme is used for Proxmox VE

hosts which were installed before the 5.0 release. When upgrading to 5.0, the names are kept as-is.

• Bridge names: vmbr[N], where 0 ≤N≤4094 (vmbr0 —vmbr4094)

• Bonds: bond[N], where 0 ≤N (bond0,bond1,…)

• VLANs: Simply add the VLAN number to the device name, separated by a period (eno1.50,bond1.

30)

This makes it easier to debug networks problems, because the device name implies the device type.

Systemd Network Interface Names

Systemd uses the two character prefix en for Ethernet network devices. The next characters depends on the

device driver and the fact which schema matches first.

• o<index>[n<phys_port_name>|d<dev_port>] — devices on board

• s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id

• [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id

• x<MAC> — device by MAC address

The most common patterns are:

• eno1 — is the first on board NIC

• enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1.

For more information see Predictable Network Interface Names.

Proxmox VE Administration Guide 22 / 336

3.3.2 Choosing a network configuration

Depending on your current network organization and your resources you can choose either a bridged, routed,

or masquerading networking setup.

Proxmox VE server in a private LAN, using an external gateway to reach the internet

The Bridged model makes the most sense in this case, and this is also the default mode on new Proxmox

VE installations. Each of your Guest system will have a virtual interface attached to the Proxmox VE bridge.

This is similar in effect to having the Guest network card directly connected to a new switch on your LAN, the

Proxmox VE host playing the role of the switch.

Proxmox VE server at hosting provider, with public IP ranges for Guests

For this setup, you can use either a Bridged or Routed model, depending on what your provider allows.

Proxmox VE server at hosting provider, with a single public IP address

In that case the only way to get outgoing network accesses for your guest systems is to use Masquerading.

For incoming network access to your guests, you will need to configure Port Forwarding.

For further flexibility, you can configure VLANs (IEEE 802.1q) and network bonding, also known as «link

aggregation». That way it is possible to build complex and flexible virtual networks.

3.3.3 Default Configuration using a Bridge

Bridges are like physical network switches implemented in software. All VMs can share a single bridge, or

you can create multiple bridges to separate network domains. Each host can have up to 4094 bridges.

The installation program creates a single bridge named vmbr0, which is connected to the first Ethernet

card. The corresponding configuration in /etc/network/interfaces might look like this:

auto lo

iface lo inet loopback

iface eno1 inet manual

auto vmbr0

iface vmbr0 inet static

address 192.168.10.2

netmask 255.255.255.0

gateway 192.168.10.1

bridge_ports eno1

bridge_stp off

bridge_fd 0

Virtual machines behave as if they were directly connected to the physical network. The network, in turn,

sees each virtual machine as having its own MAC, even though there is only one network cable connecting

all of these VMs to the network.

Proxmox VE Administration Guide 23 / 336

3.3.4 Routed Configuration

Most hosting providers do not support the above setup. For security reasons, they disable networking as

soon as they detect multiple MAC addresses on a single interface.

Tip

Some providers allows you to register additional MACs on there management interface. This avoids the

problem, but is clumsy to configure because you need to register a MAC for each of your VMs.

You can avoid the problem by “routing” all traffic via a single interface. This makes sure that all network

packets use the same MAC address.

A common scenario is that you have a public IP (assume 198.51.100.5 for this example), and an addi-

tional IP block for your VMs (203.0.113.16/29). We recommend the following setup for such situations:

auto lo

iface lo inet loopback

auto eno1

iface eno1 inet static

address 198.51.100.5

netmask 255.255.255.0

gateway 198.51.100.1

post-up echo 1 > /proc/sys/net/ipv4/ip_forward

post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp

auto vmbr0

iface vmbr0 inet static

address 203.0.113.17

netmask 255.255.255.248

bridge_ports none

bridge_stp off

bridge_fd 0

3.3.5 Masquerading (NAT) with iptables

Masquerading allows guests having only a private IP address to access the network by using the host IP

address for outgoing traffic. Each outgoing packet is rewritten by iptables to appear as originating from

the host, and responses are rewritten accordingly to be routed to the original sender.

auto lo

iface lo inet loopback

auto eno1

#real IP address

iface eno1 inet static

address 198.51.100.5

netmask 255.255.255.0

gateway 198.51.100.1

Proxmox VE Administration Guide 24 / 336

auto vmbr0

#private sub network

iface vmbr0 inet static

address 10.10.10.1

netmask 255.255.255.0

bridge_ports none

bridge_stp off

bridge_fd 0

post-up echo 1 > /proc/sys/net/ipv4/ip_forward

post-up iptables -t nat -A POSTROUTING -s ’10.10.10.0/24’ -o eno1 ←—

-j MASQUERADE

post-down iptables -t nat -D POSTROUTING -s ’10.10.10.0/24’ -o eno1 ←—

-j MASQUERADE

3.3.6 Linux Bond

Bonding (also called NIC teaming or Link Aggregation) is a technique for binding multiple NIC’s to a single

network device. It is possible to achieve different goals, like make the network fault-tolerant, increase the

performance or both together.

High-speed hardware like Fibre Channel and the associated switching hardware can be quite expensive. By

doing link aggregation, two NICs can appear as one logical interface, resulting in double speed. This is a

native Linux kernel feature that is supported by most switches. If your nodes have multiple Ethernet ports,

you can distribute your points of failure by running network cables to different switches and the bonded

connection will failover to one cable or the other in case of network trouble.

Aggregated links can improve live-migration delays and improve the speed of replication of data between

Proxmox VE Cluster nodes.

There are 7 modes for bonding:

•Round-robin (balance-rr): Transmit network packets in sequential order from the first available network

interface (NIC) slave through the last. This mode provides load balancing and fault tolerance.

•Active-backup (active-backup): Only one NIC slave in the bond is active. A different slave becomes

active if, and only if, the active slave fails. The single logical bonded interface’s MAC address is externally

visible on only one NIC (port) to avoid distortion in the network switch. This mode provides fault tolerance.

•XOR (balance-xor): Transmit network packets based on [(source MAC address XOR’d with destination

MAC address) modulo NIC slave count]. This selects the same NIC slave for each destination MAC

address. This mode provides load balancing and fault tolerance.

•Broadcast (broadcast): Transmit network packets on all slave network interfaces. This mode provides

fault tolerance.

•IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP): Creates aggregation groups that share the

same speed and duplex settings. Utilizes all slave network interfaces in the active aggregator group ac-

cording to the 802.3ad specification.

•Adaptive transmit load balancing (balance-tlb): Linux bonding driver mode that does not require any

special network-switch support. The outgoing network packet traffic is distributed according to the current

Proxmox VE Administration Guide 25 / 336

load (computed relative to the speed) on each network interface slave. Incoming traffic is received by one

currently designated slave network interface. If this receiving slave fails, another slave takes over the MAC

address of the failed receiving slave.

•Adaptive load balancing (balance-alb): Includes balance-tlb plus receive load balancing (rlb) for IPV4

traffic, and does not require any special network switch support. The receive load balancing is achieved by

ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out

and overwrites the source hardware address with the unique hardware address of one of the NIC slaves in

the single logical bonded interface such that different network-peers use different MAC addresses for their

network packet traffic.

If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using the corresponding

bonding mode (802.3ad). Otherwise you should generally use the active-backup mode.

If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode

on the bonding interfaces, other modes are unsupported.

The following bond configuration can be used as distributed/shared storage network. The benefit would be

that you get more speed and the network will be fault-tolerant.

Example: Use bond with fixed IP address

auto lo

iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0

iface bond0 inet static

slaves eno1 eno2

address 192.168.1.2

netmask 255.255.255.0

bond_miimon 100

bond_mode 802.3ad

bond_xmit_hash_policy layer2+3

auto vmbr0

iface vmbr0 inet static

address 10.10.10.2

netmask 255.255.255.0

gateway 10.10.10.1

bridge_ports eno1

bridge_stp off

bridge_fd 0

Another possibility it to use the bond directly as bridge port. This can be used to make the guest network

fault-tolerant.

Example: Use a bond as bridge port

Proxmox VE Administration Guide 26 / 336

auto lo

iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0

iface bond0 inet manual

slaves eno1 eno2

bond_miimon 100

bond_mode 802.3ad

bond_xmit_hash_policy layer2+3

auto vmbr0

iface vmbr0 inet static

address 10.10.10.2

netmask 255.255.255.0

gateway 10.10.10.1

bridge_ports bond0

bridge_stp off

bridge_fd 0

3.3.7 VLAN 802.1Q

A virtual LAN (VLAN) is a broadcast domain that is partitioned and isolated in the network at layer two. So it

is possible to have multiple networks (4096) in a physical network, each independent of the other ones.

Each VLAN network is identified by a number often called tag. Network packages are then tagged to identify

which virtual network they belong to.

VLAN for Guest Networks

Proxmox VE supports this setup out of the box. You can specify the VLAN tag when you create a VM.

The VLAN tag is part of the guest network confinuration. The networking layer supports differnet modes to

implement VLANs, depending on the bridge configuration:

•VLAN awareness on the Linux bridge: In this case, each guest’s virtual network card is assigned to

a VLAN tag, which is transparently supported by the Linux bridge. Trunk mode is also possible, but that

makes the configuration in the guest necessary.

•«traditional» VLAN on the Linux bridge: In contrast to the VLAN awareness method, this method is not

transparent and creates a VLAN device with associated bridge for each VLAN. That is, if e.g. in our default

network, a guest VLAN 5 is used to create eno1.5 and vmbr0v5, which remains until rebooting.

•Open vSwitch VLAN: This mode uses the OVS VLAN feature.

•Guest configured VLAN: VLANs are assigned inside the guest. In this case, the setup is completely

done inside the guest and can not be influenced from the outside. The benefit is that you can use more

than one VLAN on a single virtual NIC.

Proxmox VE Administration Guide 27 / 336

VLAN on the Host

To allow host communication with an isolated network. It is possible to apply VLAN tags to any network

device (NIC, Bond, Bridge). In general, you should configure the VLAN on the interface with the least

abstraction layers between itself and the physical NIC.

For example, in a default configuration where you want to place the host management address on a separate

VLAN.

Note

In the examples we use the VLAN at bridge level to ensure the correct function of VLAN 5 in the guest

network, but in combination with VLAN anwareness bridge this it will not work for guest network VLAN 5.

The downside of this setup is more CPU usage.

Example: Use VLAN 5 for the Proxmox VE management IP

auto lo

iface lo inet loopback

iface eno1 inet manual

iface eno1.5 inet manual

auto vmbr0v5

iface vmbr0v5 inet static

address 10.10.10.2

netmask 255.255.255.0

gateway 10.10.10.1

bridge_ports eno1.5

bridge_stp off

bridge_fd 0

auto vmbr0

iface vmbr0 inet manual

bridge_ports eno1

bridge_stp off

bridge_fd 0

The next example is the same setup but a bond is used to make this network fail-safe.

Example: Use VLAN 5 with bond0 for the Proxmox VE management IP

auto lo

iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0

Proxmox VE Administration Guide 28 / 336

iface bond0 inet manual

slaves eno1 eno2

bond_miimon 100

bond_mode 802.3ad

bond_xmit_hash_policy layer2+3

iface bond0.5 inet manual

auto vmbr0v5

iface vmbr0v5 inet static

address 10.10.10.2

netmask 255.255.255.0

gateway 10.10.10.1

bridge_ports bond0.5

bridge_stp off

bridge_fd 0

auto vmbr0

iface vmbr0 inet manual

bridge_ports bond0

bridge_stp off

bridge_fd 0

3.4 Time Synchronization

The Proxmox VE cluster stack itself relies heavily on the fact that all the nodes have precisely synchronized

time. Some other components, like Ceph, also refuse to work properly if the local time on nodes is not in

sync.

Time synchronization between nodes can be achieved with the “Network Time Protocol” (NTP). Proxmox VE

uses systemd-timesyncd as NTP client by default, preconfigured to use a set of public servers. This

setup works out of the box in most cases.

3.4.1 Using Custom NTP Servers

In some cases, it might be desired to not use the default NTP servers. For example, if your Proxmox VE

nodes do not have access to the public internet (e.g., because of restrictive firewall rules), you need to setup

local NTP servers and tell systemd-timesyncd to use them:

File /etc/systemd/timesyncd.conf

[Time]

NTP=ntp1.example.com ntp2.example.com ntp3.example.com ntp4.example.com

After restarting the synchronization service (systemctl restart systemd-timesyncd) you should

verify that your newly configured NTP servers are used by checking the journal (journalctl —since

-1h -u systemd-timesyncd):

Proxmox VE Administration Guide 29 / 336

…

Oct 07 14:58:36 node1 systemd[1]: Stopping Network Time Synchronization…

Oct 07 14:58:36 node1 systemd[1]: Starting Network Time Synchronization…

Oct 07 14:58:36 node1 systemd[1]: Started Network Time Synchronization.

Oct 07 14:58:36 node1 systemd-timesyncd[13514]: Using NTP server ←—

10.0.0.1:123 (ntp1.example.com).

Oct 07 14:58:36 nora systemd-timesyncd[13514]: interval/delta/delay/jitter/ ←—

drift 64s/-0.002s/0.020s/0.000s/-31ppm

…

3.5 External Metric Server

Starting with Proxmox VE 4.0, you can define external metric servers, which will be sent various stats about

your hosts, virtual machines and storages.

Currently supported are:

• graphite (see http://graphiteapp.org )

• influxdb (see https://www.influxdata.com/time-series-platform/influxdb/ )

The server definitions are saved in /etc/pve/status.cfg

3.5.1 Graphite server configuration

The definition of a server is:

graphite:

server your-server

port your-port

path your-path

where your-port defaults to 2003 and your-path defaults to proxmox

Proxmox VE sends the data over udp, so the graphite server has to be configured for this

3.5.2 Influxdb plugin configuration

The definition is:

influxdb:

server your-server

port your-port

Proxmox VE sends the data over udp, so the influxdb server has to be configured for this

Here is an example configuration for influxdb (on your influxdb server):

Proxmox VE Administration Guide 30 / 336

[[udp]]

enabled = true

bind-address = «0.0.0.0:8089»

database = «proxmox»

batch-size = 1000

batch-timeout = «1s»

With this configuration, your server listens on all IP addresses on port 8089, and writes the data in the

proxmox database

3.6 Disk Health Monitoring

Although a robust and redundant storage is recommended, it can be very helpful to monitor the health of

your local disks.

Starting with Proxmox VE 4.3, the package smartmontools 1is installed and required. This is a set of tools

to monitor and control the S.M.A.R.T. system for local hard disks.

You can get the status of a disk by issuing the following command:

# smartctl -a /dev/sdX

where /dev/sdX is the path to one of your local disks.

If the output says:

SMART support is: Disabled

you can enable it with the command:

# smartctl -s on /dev/sdX

For more information on how to use smartctl, please see man smartctl.

By default, smartmontools daemon smartd is active and enabled, and scans the disks under /dev/sdX and

/dev/hdX every 30 minutes for errors and warnings, and sends an e-mail to root if it detects a problem.

For more information about how to configure smartd, please see man smartd and man smartd.conf.

If you use your hard disks with a hardware raid controller, there are most likely tools to monitor the disks in

the raid array and the array itself. For more information about this, please refer to the vendor of your raid

controller.

3.7 Logical Volume Manager (LVM)

Most people install Proxmox VE directly on a local disk. The Proxmox VE installation CD offers several

options for local disk management, and the current default setup uses LVM. The installer let you select a

single disk for such setup, and uses that disk as physical volume for the Volume Group (VG) pve. The

following output is from a test installation using a small 8GB disk:

1smartmontools homepage https://www.smartmontools.org

Proxmox VE Administration Guide 31 / 336

# pvs

PV VG Fmt Attr PSize PFree

/dev/sda3 pve lvm2 a— 7.87g 876.00m

# vgs

VG #PV #LV #SN Attr VSize VFree

pve 1 3 0 wz—n- 7.87g 876.00m

The installer allocates three Logical Volumes (LV) inside this VG:

# lvs

LV VG Attr LSize Pool Origin Data% Meta%

data pve twi-a-tz— 4.38g 0.00 0.63

root pve -wi-ao—- 1.75g

swap pve -wi-ao—- 896.00m

root

Formatted as ext4, and contains the operation system.

swap

Swap partition

data

This volume uses LVM-thin, and is used to store VM images. LVM-thin is preferable for this task,

because it offers efficient support for snapshots and clones.

For Proxmox VE versions up to 4.1, the installer creates a standard logical volume called “data”, which is

mounted at /var/lib/vz.

Starting from version 4.2, the logical volume “data” is a LVM-thin pool, used to store block based guest

images, and /var/lib/vz is simply a directory on the root file system.

3.7.1 Hardware

We highly recommend to use a hardware RAID controller (with BBU) for such setups. This increases perfor-

mance, provides redundancy, and make disk replacements easier (hot-pluggable).

LVM itself does not need any special hardware, and memory requirements are very low.

3.7.2 Bootloader

We install two boot loaders by default. The first partition contains the standard GRUB boot loader. The

second partition is an EFI System Partition (ESP), which makes it possible to boot on EFI systems.

3.7.3 Creating a Volume Group

Let’s assume we have an empty disk /dev/sdb, onto which we want to create a volume group named

“vmdata”.

Proxmox VE Administration Guide 32 / 336

Caution

Please note that the following commands will destroy all existing data on /dev/sdb.

First create a partition.

# sgdisk -N 1 /dev/sdb

Create a Physical Volume (PV) without confirmation and 250K metadatasize.

# pvcreate —metadatasize 250k -y -ff /dev/sdb1

Create a volume group named “vmdata” on /dev/sdb1

# vgcreate vmdata /dev/sdb1

3.7.4 Creating an extra LV for /var/lib/vz

This can be easily done by creating a new thin LV.

# lvcreate -n <Name> -V <Size[M,G,T]> <VG>/<LVThin_pool>

A real world example:

# lvcreate -n vz -V 10G pve/data

Now a filesystem must be created on the LV.

# mkfs.ext4 /dev/pve/vz

At last this has to be mounted.

Warning

be sure that /var/lib/vz is empty. On a default installation it’s not.

To make it always accessible add the following line in /etc/fstab.

# echo ’/dev/pve/vz /var/lib/vz ext4 defaults 0 2’ >> /etc/fstab

3.7.5 Resizing the thin pool

Resize the LV and the metadata pool can be achieved with the following command.

# lvresize —size +<size[\M,G,T]> —poolmetadatasize +<size[\M,G]> < ←—

VG>/<LVThin_pool>

Note

When extending the data pool, the metadata pool must also be extended.

Proxmox VE Administration Guide 33 / 336

3.7.6 Create a LVM-thin pool

A thin pool has to be created on top of a volume group. How to create a volume group see Section LVM.

# lvcreate -L 80G -T -n vmstore vmdata

3.8 ZFS on Linux

ZFS is a combined file system and logical volume manager designed by Sun Microsystems. Starting with

Proxmox VE 3.4, the native Linux kernel port of the ZFS file system is introduced as optional file system and

also as an additional selection for the root file system. There is no need for manually compile ZFS modules

— all packages are included.

By using ZFS, its possible to achieve maximum enterprise features with low budget hardware, but also high

performance systems by leveraging SSD caching or even SSD only setups. ZFS can replace cost intense

hardware raid cards by moderate CPU and memory load combined with easy management.

GENERAL ZFS ADVANTAGES

• Easy configuration and management with Proxmox VE GUI and CLI.

• Reliable

• Protection against data corruption

• Data compression on file system level

• Snapshots

• Copy-on-write clone

• Various raid levels: RAID0, RAID1, RAID10, RAIDZ-1, RAIDZ-2 and RAIDZ-3

• Can use SSD for cache

• Self healing

• Continuous integrity checking

• Designed for high storage capacities

• Protection against data corruption

• Asynchronous replication over network

• Open Source

• Encryption

• . . .

Proxmox VE Administration Guide 34 / 336

3.8.1 Hardware

ZFS depends heavily on memory, so you need at least 8GB to start. In practice, use as much you can get

for your hardware/budget. To prevent data corruption, we recommend the use of high quality ECC RAM.

If you use a dedicated cache and/or log disk, you should use an enterprise class SSD (e.g. Intel SSD DC

S3700 Series). This can increase the overall performance significantly.

Important

Do not use ZFS on top of hardware controller which has its own cache management. ZFS needs to

directly communicate with disks. An HBA adapter is the way to go, or something like LSI controller

flashed in “IT” mode.

If you are experimenting with an installation of Proxmox VE inside a VM (Nested Virtualization), don’t use

virtio for disks of that VM, since they are not supported by ZFS. Use IDE or SCSI instead (works also

with virtio SCSI controller type).

3.8.2 Installation as Root File System

When you install using the Proxmox VE installer, you can choose ZFS for the root file system. You need to

select the RAID type at installation time:

RAID0 Also called “striping”. The capacity of such volume is the sum of the capacities of all

disks. But RAID0 does not add any redundancy, so the failure of a single drive

makes the volume unusable.

RAID1 Also called “mirroring”. Data is written identically to all disks. This mode requires at

least 2 disks with the same size. The resulting capacity is that of a single disk.

RAID10 A combination of RAID0 and RAID1. Requires at least 4 disks.

RAIDZ-1 A variation on RAID-5, single parity. Requires at least 3 disks.

RAIDZ-2 A variation on RAID-5, double parity. Requires at least 4 disks.

RAIDZ-3 A variation on RAID-5, triple parity. Requires at least 5 disks.

The installer automatically partitions the disks, creates a ZFS pool called rpool, and installs the root file

system on the ZFS subvolume rpool/ROOT/pve-1.

Another subvolume called rpool/data is created to store VM images. In order to use that with the

Proxmox VE tools, the installer creates the following configuration entry in /etc/pve/storage.cfg:

zfspool: local-zfs

pool rpool/data

sparse

content images,rootdir

After installation, you can view your ZFS pool status using the zpool command:

Proxmox VE Administration Guide 35 / 336

# zpool status

pool: rpool

state: ONLINE

scan: none requested

config:

NAME STATE READ WRITE CKSUM

rpool ONLINE 0 0 0

mirror-0 ONLINE 0 0 0

sda2 ONLINE 0 0 0

sdb2 ONLINE 0 0 0

mirror-1 ONLINE 0 0 0

sdc ONLINE 0 0 0

sdd ONLINE 0 0 0

errors: No known data errors

The zfs command is used configure and manage your ZFS file systems. The following command lists all

file systems after installation:

# zfs list

NAME USED AVAIL REFER MOUNTPOINT

rpool 4.94G 7.68T 96K /rpool

rpool/ROOT 702M 7.68T 96K /rpool/ROOT

rpool/ROOT/pve-1 702M 7.68T 702M /

rpool/data 96K 7.68T 96K /rpool/data

rpool/swap 4.25G 7.69T 64K —

3.8.3 Bootloader

The default ZFS disk partitioning scheme does not use the first 2048 sectors. This gives enough room to

install a GRUB boot partition. The Proxmox VE installer automatically allocates that space, and installs the

GRUB boot loader there. If you use a redundant RAID setup, it installs the boot loader on all disk required

for booting. So you can boot even if some disks fail.

Note

It is not possible to use ZFS as root file system with UEFI boot.

3.8.4 ZFS Administration

This section gives you some usage examples for common tasks. ZFS itself is really powerful and provides

many options. The main commands to manage ZFS are zfs and zpool. Both commands come with great

manual pages, which can be read with:

# man zpool

# man zfs

Proxmox VE Administration Guide 36 / 336

Create a new zpool

To create a new pool, at least one disk is needed. The ashift should have the same sector-size (2 power

of ashift) or larger as the underlying disk.

zpool create -f -o ashift=12 <pool> <device>

To activate compression

zfs set compression=lz4 <pool>

Create a new pool with RAID-0

Minimum 1 Disk

zpool create -f -o ashift=12 <pool> <device1> <device2>

Create a new pool with RAID-1

Minimum 2 Disks

zpool create -f -o ashift=12 <pool> mirror <device1> <device2>

Create a new pool with RAID-10

Minimum 4 Disks

zpool create -f -o ashift=12 <pool> mirror <device1> <device2> ←—

mirror <device3> <device4>

Create a new pool with RAIDZ-1

Minimum 3 Disks

zpool create -f -o ashift=12 <pool> raidz1 <device1> <device2> < ←—

device3>

Create a new pool with RAIDZ-2

Minimum 4 Disks

zpool create -f -o ashift=12 <pool> raidz2 <device1> <device2> < ←—

device3> <device4>

Create a new pool with cache (L2ARC)

It is possible to use a dedicated cache drive partition to increase the performance (use SSD).

As <device> it is possible to use more devices, like it’s shown in «Create a new pool with RAID*».

zpool create -f -o ashift=12 <pool> <device> cache <cache_device>

Proxmox VE Administration Guide 37 / 336

Create a new pool with log (ZIL)

It is possible to use a dedicated cache drive partition to increase the performance(SSD).

As <device> it is possible to use more devices, like it’s shown in «Create a new pool with RAID*».

zpool create -f -o ashift=12 <pool> <device> log <log_device>

Add cache and log to an existing pool

If you have an pool without cache and log. First partition the SSD in 2 partition with parted or gdisk

Important

Always use GPT partition tables.

The maximum size of a log device should be about half the size of physical memory, so this is usually quite

small. The rest of the SSD can be used as cache.

zpool add -f <pool> log <device-part1> cache <device-part2>

Changing a failed device

zpool replace -f <pool> <old device> <new-device>

3.8.5 Activate E-Mail Notification

ZFS comes with an event daemon, which monitors events generated by the ZFS kernel module. The daemon

can also send emails on ZFS events like pool errors. Newer ZFS packages ships the daemon in a separate

package, and you can install it using apt-get:

# apt-get install zfs-zed

To activate the daemon it is necessary to edit /etc/zfs/zed.d/zed.rc with your favourite editor, and

uncomment the ZED_EMAIL_ADDR setting:

ZED_EMAIL_ADDR=»root»

Please note Proxmox VE forwards mails to root to the email address configured for the root user.

Important

The only setting that is required is ZED_EMAIL_ADDR. All other settings are optional.

Proxmox VE Administration Guide 38 / 336

3.8.6 Limit ZFS Memory Usage

It is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to prevent

performance shortage of the host. Use your preferred editor to change the configuration in /etc/modpr

obe.d/zfs.conf and insert:

options zfs zfs_arc_max=8589934592

This example setting limits the usage to 8GB.

Important

If your root file system is ZFS you must update your initramfs every time this value changes:

update-initramfs -u

SWAP on ZFS

SWAP on ZFS on Linux may generate some troubles, like blocking the server or generating a high IO load,

often seen when starting a Backup to an external Storage.

We strongly recommend to use enough memory, so that you normally do not run into low memory situations.

Additionally, you can lower the “swappiness” value. A good value for servers is 10:

sysctl -w vm.swappiness=10

To make the swappiness persistent, open /etc/sysctl.conf with an editor of your choice and add the

following line:

vm.swappiness = 10

Table 3.1: Linux kernel swappiness parameter values

Value Strategy

vm.swappiness = 0 The kernel will swap only to avoid an out of memory condition

vm.swappiness = 1 Minimum amount of swapping without disabling it entirely.

vm.swappiness = 10 This value is sometimes recommended to improve performance

when sufficient memory exists in a system.

vm.swappiness = 60 The default value.

vm.swappiness = 100 The kernel will swap aggressively.

Proxmox VE Administration Guide 39 / 336

3.9 Certificate Management

3.9.1 Certificates for communication within the cluster

Each Proxmox VE cluster creates its own internal Certificate Authority (CA) and generates a self-signed cer-

tificate for each node. These certificates are used for encrypted communication with the cluster’s pveproxy

service and the Shell/Console feature if SPICE is used.

The CA certificate and key are stored in the Proxmox Cluster File System (pmxcfs) Chapter 7.

3.9.2 Certificates for API and web GUI

The REST API and web GUI are provided by the pveproxy service, which runs on each node.

You have the following options for the certificate used by pveproxy:

1. By default the node-specific certificate in /etc/pve/nodes/NODENAME/pve-ssl.pem is used.

This certificate is signed by the cluster CA and therefore not trusted by browsers and operating sys-

tems by default.

2. use an externally provided certificate (e.g. signed by a commercial CA).

3. use ACME (e.g., Let’s Encrypt) to get a trusted certificate with automatic renewal.

For options 2 and 3 the file /etc/pve/local/pveproxy-ssl.pem (and /etc/pve/local/pve

proxy-ssl.key, which needs to be without password) is used.

Certificates are managed with the Proxmox VE Node management command (see the pvenode(1) man-

page).

Warning

Do not replace or manually modify the automatically generated node certificate files in /etc/

pve/local/pve-ssl.pem and /etc/pve/local/pve-ssl.key or the cluster CA files

in /etc/pve/pve-root-ca.pem and /etc/pve/priv/pve-root-ca.key.

Getting trusted certificates via ACME

Proxmox VE includes an implementation of the Automatic Certificate Management Environment ACME pro-

tocol, allowing Proxmox VE admins to interface with Let’s Encrypt for easy setup of trusted TLS certificates

which are accepted out of the box on most modern operating systems and browsers.

Currently the two ACME endpoints implemented are Let’s Encrypt (LE) and its staging environment (see

https://letsencrypt.org), both using the standalone HTTP challenge.

Because of rate-limits you should use LE staging for experiments.

There are a few prerequisites to use Let’s Encrypt:

1. Port 80 of the node needs to be reachable from the internet.

2. There must be no other listener on port 80.

Proxmox VE Administration Guide 40 / 336

3. The requested (sub)domain needs to resolve to a public IP of the Node.

4. You have to accept the ToS of Let’s Encrypt.

At the moment the GUI uses only the default ACME account.

Example: Sample pvenode invocation for using Let’s Encrypt certificates

root@proxmox:~# pvenode acme account register default mail@example.invalid

Directory endpoints:

0) Let’s Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)

1) Let’s Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/ ←—

directory)

2) Custom

Enter selection:

1

Attempting to fetch Terms of Service from ’https://acme-staging-v02.api. ←—

letsencrypt.org/directory’..

Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November ←—

-15-2017.pdf

Do you agree to the above terms? [y|N]y

Attempting to register account with ’https://acme-staging-v02.api. ←—

letsencrypt.org/directory’..

Generating ACME account key..

Registering ACME account..

Registration successful, account URL: ’https://acme-staging-v02.api. ←—

letsencrypt.org/acme/acct/xxxxxxx’

Task OK

root@proxmox:~# pvenode acme account list

default

root@proxmox:~# pvenode config set —acme domains=example.invalid

root@proxmox:~# pvenode acme cert order

Loading ACME account details

Placing ACME order

Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/ ←—

xxxxxxxxxxxxxx

Getting authorization details from

’https://acme-staging-v02.api.letsencrypt.org/acme/authz/ ←—

xxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxx-xxxxxxx’

… pending!

Setting up webserver

Triggering validation

Sleeping for 5 seconds

Status is ’valid’!

All domains validated!

Creating CSR

Finalizing order

Proxmox VE Administration Guide 41 / 336

Checking order status

valid!

Downloading certificate

Setting pveproxy certificate and key

Restarting pveproxy

Task OK

Switching from the staging to the regular ACME directory

Changing the ACME directory for an account is unsupported. If you want to switch an account from the

staging ACME directory to the regular, trusted, one you need to deactivate it and recreate it.

This procedure is also needed to change the default ACME account used in the GUI.

Example: Changing the default ACME account from the staging to the regular directory

root@proxmox:~# pvenode acme account info default

Directory URL: https://acme-staging-v02.api.letsencrypt.org/directory

Account URL: https://acme-staging-v02.api.letsencrypt.org/acme/acct/6332194

Terms Of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November ←—

-15-2017.pdf

Account information:

ID: xxxxxxx

Contact:

— mailto:example@proxmox.com

Creation date: 2018-07-31T08:41:44.54196435Z

Initial IP: 192.0.2.1

Status: valid

root@proxmox:~# pvenode acme account deactivate default

Renaming account file from ’/etc/pve/priv/acme/default’ to ’/etc/pve/priv/ ←—

acme/_deactivated_default_4’

Task OK

root@proxmox:~# pvenode acme account register default example@proxmox.com

Directory endpoints:

0) Let’s Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)

1) Let’s Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/ ←—

directory)

2) Custom

Enter selection:

0

Attempting to fetch Terms of Service from ’https://acme-v02.api.letsencrypt ←—

.org/directory’..

Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November ←—

-15-2017.pdf

Do you agree to the above terms? [y|N]y

Proxmox VE Administration Guide 42 / 336

Attempting to register account with ’https://acme-v02.api.letsencrypt.org/ ←—

directory’..

Generating ACME account key..

Registering ACME account..

Registration successful, account URL: ’https://acme-v02.api.letsencrypt.org ←—

/acme/acct/39335247’

Task OK

Automatic renewal of ACME certificates

If a node has been successfully configured with an ACME-provided certificate (either via pvenode or via the

GUI), the certificate will be automatically renewed by the pve-daily-update.service. Currently, renewal will be

attempted if the certificate has expired or will expire in the next 30 days.

Proxmox VE Administration Guide 43 / 336

Chapter 4

Hyper-converged Infrastructure

Proxmox VE is a virtualization platform that tightly integrates compute, storage and networking resources,

manages highly available clusters, backup/restore as well as disaster recovery. All components are software-

defined and compatible with one another.

Therefore it is possible to administrate them like a single system via the centralized web management inter-

face. These capabilities make Proxmox VE an ideal choice to deploy and manage an open source hyper-

converged infrastructure.

4.1 Benefits of a Hyper-Converged Infrastructure (HCI) with Proxmox

VE

A hyper-converged infrastructure is especially useful for deployments in which a high infrastructure demand

meets a low administration budget, for distributed setups such as remote and branch office environments or

for virtual private and public clouds.

HCI provides the following advantages:

• Scalability: seamless expansion of compute, network and storage devices (i.e. scale up servers and

storage quickly and independently from each other).

• Low cost: Proxmox VE is open source and integrates all components you need such as compute, storage,

networking, backup, and management center. It can replace an expensive compute/storage infrastructure.

• Data protection and efficiency: services such as backup and disaster recovery are integrated.

• Simplicity: easy configuration and centralized administration.

• Open Source: No vendor lock-in.

Proxmox VE Administration Guide 44 / 336

4.2 Manage Ceph Services on Proxmox VE Nodes

Proxmox VE unifies your compute and storage systems, i.e. you can use the same physical nodes within a

cluster for both computing (processing VMs and containers) and replicated storage. The traditional silos of

compute and storage resources can be wrapped up into a single hyper-converged appliance. Separate stor-

age networks (SANs) and connections via network attached storages (NAS) disappear. With the integration

of Ceph, an open source software-defined storage platform, Proxmox VE has the ability to run and manage

Ceph storage directly on the hypervisor nodes.

Ceph is a distributed object store and file system designed to provide excellent performance, reliability and

scalability.

SOME ADVANTAGES OF CEPH ON PROXMOX VE ARE:

• Easy setup and management with CLI and GUI support

• Thin provisioning

• Snapshots support

• Self healing

• Scalable to the exabyte level

Proxmox VE Administration Guide 45 / 336

• Setup pools with different performance and redundancy characteristics

• Data is replicated, making it fault tolerant

• Runs on economical commodity hardware

• No need for hardware RAID controllers

• Open source

For small to mid sized deployments, it is possible to install a Ceph server for RADOS Block Devices (RBD)

directly on your Proxmox VE cluster nodes, see Ceph RADOS Block Devices (RBD) Section 8.14. Recent

hardware has plenty of CPU power and RAM, so running storage services and VMs on the same node is

possible.

To simplify management, we provide pveceph — a tool to install and manage Ceph services on Proxmox VE

nodes.

CEPH CONSISTS OF A COUPLE OF DAEMONS 1,FOR USE AS A RBD STORAGE:

• Ceph Monitor (ceph-mon)

• Ceph Manager (ceph-mgr)

• Ceph OSD (ceph-osd; Object Storage Daemon)

Tip

We recommend to get familiar with the Ceph vocabulary. a

aCeph glossary http://docs.ceph.com/docs/luminous/glossary

4.2.1 Precondition

To build a Proxmox Ceph Cluster there should be at least three (preferably) identical servers for the setup.

A 10Gb network, exclusively used for Ceph, is recommended. A meshed network setup is also an option if

there are no 10Gb switches available, see our wiki article 2.

Check also the recommendations from Ceph’s website.

Avoid RAID

As Ceph handles data object redundancy and multiple parallel writes to disks (OSDs) on its own, using a

RAID controller normally doesn’t improve performance or availability. On the contrary, Ceph is designed to

handle whole disks on it’s own, without any abstraction in between. RAID controller are not designed for

the Ceph use case and may complicate things and sometimes even reduce performance, as their write and

caching algorithms may interfere with the ones from Ceph.

Warning

Avoid RAID controller, use host bus adapter (HBA) instead.

2Full Mesh Network for Ceph https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server

Proxmox VE Administration Guide 46 / 336

4.2.2 Installation of Ceph Packages

On each node run the installation script as follows:

pveceph install

This sets up an apt package repository in /etc/apt/sources.list.d/ceph.list and installs

the required software.

4.2.3 Creating initial Ceph configuration

After installation of packages, you need to create an initial Ceph configuration on just one node, based on

your network (10.10.10.0/24 in the following example) dedicated for Ceph:

pveceph init —network 10.10.10.0/24

This creates an initial configuration at /etc/pve/ceph.conf. That file is automatically distributed to all

Proxmox VE nodes by using pmxcfs Chapter 7. The command also creates a symbolic link from /etc/

ceph/ceph.conf pointing to that file. So you can simply run Ceph commands without the need to specify

a configuration file.

Proxmox VE Administration Guide 47 / 336

4.2.4 Creating Ceph Monitors

The Ceph Monitor (MON) 3maintains a master copy of the cluster map. For high availability you need to

have at least 3 monitors.

On each node where you want to place a monitor (three monitors are recommended), create it by using the

Ceph →Monitor tab in the GUI or run.

pveceph createmon

This will also install the needed Ceph Manager (ceph-mgr) by default. If you do not want to install a manager,

specify the -exclude-manager option.

4.2.5 Creating Ceph Manager

The Manager daemon runs alongside the monitors, providing an interface for monitoring the cluster. Since

the Ceph luminous release the ceph-mgr 4daemon is required. During monitor installation the ceph manager

will be installed as well.

3Ceph Monitor http://docs.ceph.com/docs/luminous/start/intro/

4Ceph Manager http://docs.ceph.com/docs/luminous/mgr/

Proxmox VE Administration Guide 48 / 336

Note

It is recommended to install the Ceph Manager on the monitor nodes. For high availability install more

then one manager.

pveceph createmgr

4.2.6 Creating Ceph OSDs

via GUI or via CLI as follows:

pveceph createosd /dev/sd[X]

Tip

We recommend a Ceph cluster size, starting with 12 OSDs, distributed evenly among your, at least three

nodes (4 OSDs on each node).

If the disk was used before (eg. ZFS/RAID/OSD), to remove partition table, boot sector and any OSD leftover

the following commands should be sufficient.

Proxmox VE Administration Guide 49 / 336

dd if=/dev/zero of=/dev/sd[X] bs=1M count=200

ceph-disk zap /dev/sd[X]

Warning

The above commands will destroy data on the disk!

Ceph Bluestore

Starting with the Ceph Kraken release, a new Ceph OSD storage type was introduced, the so called Blue-

store 5. This is the default when creating OSDs in Ceph luminous.

pveceph createosd /dev/sd[X]

Note

In order to select a disk in the GUI, to be more failsafe, the disk needs to have a GPT apartition table. You

can create this with gdisk /dev/sd(x). If there is no GPT, you cannot select the disk as DB/WAL.

aGPT partition table https://en.wikipedia.org/wiki/GUID_Partition_Table

If you want to use a separate DB/WAL device for your OSDs, you can specify it through the -journal_dev

option. The WAL is placed with the DB, if not specified separately.

pveceph createosd /dev/sd[X] -journal_dev /dev/sd[Y]

Note

The DB stores BlueStore’s internal metadata and the WAL is BlueStore’s internal journal or write-ahead

log. It is recommended to use a fast SSDs or NVRAM for better performance.

Ceph Filestore

Till Ceph luminous, Filestore was used as storage type for Ceph OSDs. It can still be used and might give

better performance in small setups, when backed by a NVMe SSD or similar.

pveceph createosd /dev/sd[X] -bluestore 0

Note

In order to select a disk in the GUI, the disk needs to have a GPT apartition table. You can create this with

gdisk /dev/sd(x). If there is no GPT, you cannot select the disk as journal. Currently the journal

size is fixed to 5 GB.

5Ceph Bluestore http://ceph.com/community/new-luminous-bluestore/

Proxmox VE Administration Guide 50 / 336

If you want to use a dedicated SSD journal disk:

pveceph createosd /dev/sd[X] -journal_dev /dev/sd[Y] -bluestore 0

Example: Use /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD journal disk.

pveceph createosd /dev/sdf -journal_dev /dev/sdb -bluestore 0

This partitions the disk (data and journal partition), creates filesystems and starts the OSD, afterwards it is

running and fully functional.

Note

This command refuses to initialize disk when it detects existing data. So if you want to overwrite a disk

you should remove existing data first. You can do that using: ceph-disk zap /dev/sd[X]

You can create OSDs containing both journal and data partitions or you can place the journal on a dedicated

SSD. Using a SSD journal disk is highly recommended to achieve good performance.

4.2.7 Creating Ceph Pools

Proxmox VE Administration Guide 51 / 336

A pool is a logical group for storing objects. It holds Placement Groups (PG), a collection of objects.

When no options are given, we set a default of 64 PGs, a size of 3 replicas and a min_size of 2 replicas

for serving objects in a degraded state.

Note

The default number of PGs works for 2-6 disks. Ceph throws a «HEALTH_WARNING» if you have too few

or too many PGs in your cluster.

It is advised to calculate the PG number depending on your setup, you can find the formula and the PG

calculator 6online. While PGs can be increased later on, they can never be decreased.