General Advice

Understand the OSS components that make up the Chef Server

A good way to think about the Chef server is as a collection of microservices components underpinned by OSS software:

• Nginx (openresty)
• PostgreSQL
• Solr
• RabbitMQ
• Redis
• Chef
• Erlang/OTP
• Ruby
• Runit
• The Linux Kernel
  • LVM
  • Storage subsystem
  • Network stack

It’s important to understand the performance characteristics, monitoring and troubleshooting of these components. Especially Postgres, Solr, RabbitMQ, Runit and Linux systems in general. It’s worth noting that the Chef server core is Open Source, and all if its code can be examined on Github

Because these components are glued together using Chef, it’s highly recommended that you familiarize yourself with the cookbooks that configure the Chef server when you run chef-server-ctl reconfigure

Have good monitoring in place

We don’t provide prescriptive monitoring guidance at this time, but here’s our advice:

• Use existing Open source software (Sensu, Nagios, etc) to collect metrics and test the health of the OSS components. This should be fairly straightforward to set up.
  • Use pgBadger for Postgres log analysis and pg_stat_statements
  • Install the RabbitMQ Management Plugin for detailed monitoring of RabbitMQ
• Configure your monitoring systems and load balancers to query the Health status endpoint of erchef (https://mychefserver/_status)
• Run a graphite server. erchef will send detailed statistics if you set the following in your chef-server.rb file:

  1 2 3

  folsom_graphite['enabled'] = true folsom_graphite['host'] = 'graphite.mycompany.com' folsom_graphite['port'] = 2003

• Use Splunk or Logstash to collect and analyze your Chef server logs.
  • You can collect and graph useful performance data by graphing /var/log/opscode/opscode-erchef/requests.log.N, /var/log/opscode/oc_bifrost/requests.log.N and /var/log/opscode/opscode-reporting/requests.log.N
  • Each request line will show (in ms) various performance counter. For example: req_time=20; rdbms_time=2; rdbms_count=3; authz_time=5; authz_count=1;

Think about API requests per second rather than node counts

A very common measurement for the size of Chef servers/clusters is the number of nodes they serve. However, this number is not terribly useful because of other elements that can cause very wide variation. Namely:

• The interval and splay of Chef client runs
  • 1000 nodes every hour == 500 nodes every 30 minutes
  • Insufficient splay can cause a “stampede condition” on the Chef server. Splay should be equal to the interval in order to get maximum smoothness of request load.
• The number and complexity of search requests and databag fetches performed during each Chef run
• The number of cookbooks depended on for each Chef run. More cookbooks adds loading to the depsolver and also to the Bookshelf service which serves cookbooks
• The size of node data, which we’ve seen range from 32kb to 5MB (the default maximum is 1MB but can be increased). This adds load to the indexing service (opscode-expander) as well as to Solr

Although it’s not perfect, we’ve found that a good rule of thumb for examining active Chef servers is the number of API requests per second aggregated across the entire cluster. We’ve found that clusters which sustained higher than 125 API RPS started to experience occasional errors.

DRBD: Don’t do it

In the field we’ve found that DRBD has a negative impact on performance and availability of Chef server clusters. Specifically:

• Because DRBD uses synchronous replication, a block is not considered “committed to disk” until it has been confirmed by both nodes in the cluster. This adds significant latency to each IOP.
• DRBD’s bandwidth is limited by the network throughput between the nodes. Dedicated cross-over links are not possible in all scenarios (for example VMs) which leads to low and inconsistent throughput.
• DRBD resyncs can take a very long time and greatly impact performance while running.
• Although DRBD protects against hardware failure, it does a very poor job of protecting against many classes of software failure. For example, a corrupt database is replicated whole to the other node, so failing over will not correct the system.

Beware LVM snapshots impact on performance

LVM is generally recommended for storing all Chef Server data (/var/opt/opscode in standalone/tier installs and /var/opt/opscode/drbd/data in HA installs) because it provides the ability to expand disks on the fly and create crash-consistent snapshots.

However it’s important to know that as LVM snapshots increase in size it is very detrimental to performance:

• http://www.percona.com/blog/2013/07/09/lvm-read-performance-during-snapshots/
• http://www.percona.com/blog/2009/02/05/disaster-lvm-performance-in-snapshot-mode/

Therefore it is recommend that snapshots are used to create consistent backups, but are immediately deleted after they are no longer needed.

Chef Server tuning tips

Server sizing

Chef Server frontends:

• Frontends run stateless services only (erchef, bifrost, reporting, manage) and can be scaled horizontally.
• They are almost always CPU bound, and only suffer memory or disk pressure during fault scenarios (typically because of backend issues).
• A good starting point for frontends is 4 CPU cores and 8 GB RAM. Disk on frontends does not matter.

Chef server backends:

• Backends mix a number of disk, memory and CPU bound services (Postgres, Solr, RabbitMQ, Expander)
• A good starting point for backends is 8 CPU cores and 32 GB of RAM.
• Flash-based storage is highly recommend, combined with the XFS filesystem and LVM.

chef-server.rb tuning settings

Database pooling:

In the erlang OTP process model, the number of workers is limited by the size of the database connection pool (default 20). Increasing the database pool allows for more workers, but puts added memory pressure on the database service.

In order to handle the greater number of connections, you must also increase the Postgres max_connections value. This value must consider an erchef, bifrost and reporting process connecting from each frontend, plus an extra 20% for breathing room.

Suggested values for a high-performing cluster with 4-6 frontends:

1
2
3

postgresql['max_connections'] = 1024
opscode_erchef['db_pool_size'] = 40
oc_bifrost['db_pool_size'] = 40

Erchef to bifrost http connection pool: erchef also maintains a pool of http connections to bifrost, the authz service. It’s important to raise the initial and maximum number of connections with respect to the database pool sizes.

1
2
3

oc_chef_authz['http_init_count'] = 100
oc_chef_authz['http_max_count'] = 100
oc_chef_authz['http_queue_max'] = 200

Erchef depsolver and keygen tuning: Two expensive computations that erchef must perform are the depsolver (a Ruby process which solves the cookbook dependencies) as well as the client key generator (which can be hit hard when large fleets of chef nodes are provisioned). Note that Chef 12 clients default to client-side key generation and you probably only need to adjust the keygen value if you still use Chef 11 clients.

Suggested values:

1
2
3

opscode_erchef['depsolver_worker_count'] = 4 # should equal the number of CPU cores
opscode_erchef['depsolver_timeout'] = 10000
opscode_erchef['keygen_cache_size'] = 1000

NEW IN CHEF SERVER 12.1.0: Bounded queueing for Pooler There are several upstream services who’s connections are managed by pooler: sqerl (database connection), depsolver workers and the authz pool (connections from erchef to bifrost). Currently when any of erchef’s pools are exhausted, it throws a 500 error. Chef Server 12.1 added the ability to add bounded queues to each pool which greatly reduces error rates and also reduces the need for large connection pools (which are suboptimal for Postgres).

Queueing is disabled by default, but is enabled by setting the timeout value to > 0. When using queueing, it’s recommended to use a smaller pool size matched with a queue that is 1-2x the size of the pool.

1
2
3
4
5
6
7
8
9
10
11

# erchef database pooler queue
opscode_erchef['db_pool_queue_max'] = 40
opscode_erchef['db_pooler_timeout'] = 2000

# bifrost database pooler queue
oc_bifrost['db_pooler_timeout'] = 2000
oc_bifrost['db_pool_queue_max'] = 40

# erchef depsolver queue
opscode_erchef['depsolver_pool_queue_max'] = 10
opscode_erchef['depsolver_pooler_timeout'] = 100000

Nginx cookbook caching: A new feature in Chef Server 12.0.4 is Nginx cookbook caching. This takes load off of the backend Bookshelf service by storing cookbook files in Nginx.

Suggested values:

1
2

opscode_erchef['nginx_bookshelf_caching'] = ":on"
opscode_erchef['s3_url_expiry_window_size'] = "100%"

PostgreSQL tuning: We already tune PostgreSQL memory settings to sane values based on the backend’s phyiscal RAM. For example, effective_cache_size is set to 50% of RAM, and shared_buffers to 25% of physical RAM.

To handle the heavy write load on large clusters, it is recommended to tune the checkpointer per [https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server]

Finally, the log_min_duration_statement setting is super useful for the detection and postmortem analysis of performance issues. It is equivalent to the “slow query log” in MySQL. The tuning setting below will log all of the queries that took longer than 1000ms to complete.

Suggested values:

1
2
3

postgresql['checkpoint_segments'] = 64
postgresql['checkpoint_completion_target'] = 0.9
postgresql['log_min_duration_statement'] = 1000

Solr JVM tuning: By default we compute Solr’s JVM heap size to be either 25% of system memory or 1024MB, whichever is smaller. Large chef server clusters should increase this value to smaller of 25% of system memory or 4096MB. Extremely large and busy Chef clusters run successfully with an 8GB Solr heap size.

Suggested values:

1
2

opscode_solr4['heap_size'] = 4096
opscode_solr4['new_size'] = 256

WARNING: It is not recommended to use a JVM heap_size above 8GB, unless you have in-depth knowledge of JVM tuning combined with detailed JVM monitoring

Chef Server 12 includes oc-id, the OAuth2 service that powers id.chef.io. After upgrading to this release, Chef customers can now run their own Supermarket service behind a firewall.

oc-id setup on your Chef Server:

You must be logged in to your Chef server via ssh and elevated to an admin user level for the following steps

1. Add the following setting to your /etc/opscode/chef-server.rb configuration file:

   1 2 3 4 5

   oc_id['applications'] = { 'supermarket' => { 'redirect_uri' => 'https://supermarket.mycompany.com/auth/chef_oauth2/callback' } }

2. run chef-server-ctl reconfigure

3. After the reconfigure, you will find the OAuth2 data in /etc/opscode/oc-id-applications/supermarket.json

   1 2 3 4 5 6

   { "name": "supermarket", "uid": "0bad0f2eb04e935718e081fb71e3b7bb47dc3681c81acb9968a8e1e32451d08b", "secret": "17cf1141cc971a10ce307611beda7f4dc6633bb54f1bc98d9f9ca76b9b127879", "redirect_uri": "https://supermarket.mycompany.com/auth/chef_oauth2/callback" }

Note the uid and secret values from this file, you will need them for the next stage.

You can add as many oc-id applications as you wish to the chef-server.rb configuration, it will create one file per application

Running your Private Supermarket server in Test Kitchen

Note: We will not use the community Supermarket cookbook, because at this time it installs Supermarket from source. Instead, we will us an Omnibus package to install

In the spirit of “code as documentation” I’ve provided a simple cookbook and test-kitchen configuration for testing Supermarket Omnibus packages. These packages are downloaded from https://packagecloud.io/chef/stable

1. Download a copy of the [supermarket-omnibus-cookbook]https://github.com/irvingpop/supermarket-omnibus-cookbook

   1 2	git clone https://github.com/irvingpop/supermarket-omnibus-cookbook.git supermarket-omnibus-cookbook cd supermarket-omnibus-cookbook

2. Create a .kitchen.local.yml file, to set your oc-id attributes (as captured in step 3 above)

   1 2 3 4 5 6 7 8 9 10 11

   --- suites: - name: default run_list: - recipe[supermarket-omnibus-cookbook::default] attributes: supermarket_omnibus: chef_server_url: https://chefserver.mycompany.com chef_oauth2_app_id: 0bad0f2eb04e935718e081fb71e3b7bb47dc3681c81acb9968a8e1e32451d08b chef_oauth2_secret: 17cf1141cc971a10ce307611beda7f4dc6633bb54f1bc98d9f9ca76b9b127879 chef_oauth2_verify_ssl: false

3. Install the vagrant-hostsupdater plugin, this will automatically add the names of your machines to your /etc/hosts file. This is important for oauth2, which cares about host names. The redirect_uri value you entered in to your oc-id configuration reflects this name.

   1	vagrant plugin install vagrant-hostsupdater

4. Start your Supermarket instance and test it

   1	kitchen converge default-centos-66 && kitchen verify default-centos-66

5. Go to your your Supermarket server and log in as a Chef user: https://default-centos-66

6. Upon login, you should see:

Uploading your first cookbook to Supermarket

1. Install the knife-supermarket gem. In ChefDK:

   1	chef gem install knife-supermarket

2. In your knife.rb file, add a setting for the supermarket server:

   1	knife[:supermarket_site] = 'https://default-centos-66'

3. To resolve any SSL errors, fetch and verify the Supermarket server’s SSL certificate:

   1 2	knife ssl fetch https://default-centos-66 knife ssl check https://default-centos-66

4. Upload your cookbook to Supermarket

   1	knife supermarket share mycookbook "Other"

Running Supermarket in Production

Supermarket is still in early stages and does not have official Support from Chef, HA, backup tools, etc. Although several of our key customers are running Supermarket in prod, they are doing it at their own risk.

In general we recommend that you start using small VMs, it’s easy to increase your VM size as you need it. Put your /var/opt/supermarket directory on a separate disk and use LVM so that it can be expanded.

Your Wrapper Cookbook attributes

We recommend that you use use a wrapper cookbook with role recipes to deploy Supermarket.

All of the keys under node['supermarket_omnibus'] are written out as /etc/supermarket/supermarket.json. You can add others as you see fit to override the defaults specified in the supermarket Omnibus package

1
2
3
4

default['supermarket_omnibus']['chef_server_url'] = 'https://chefserver.mycompany.com'
default['supermarket_omnibus']['chef_oauth2_app_id'] = '14dfcf186221781cff51eedd5ac1616'
default['supermarket_omnibus']['chef_oauth2_secret'] = 'a49402219627cfa6318d58b13e90aca'
default['supermarket_omnibus']['chef_oauth2_verify_ssl'] = false

Scaling the system

Supermarket is a Ruby on Rails app with a Postgres backend, and typical RoR scaling rules apply. If you wish to run Supermarket in a scale-out or HA mode, you can do this by building our your own back-end components:

• Database: Build a separate PostgreSQL 9.3+ server (or HA pair). Please note that the following Postgres extensions must be installed and loaded: pgpsql and pg_trgm
• Cookbook Storage Cookbook tarballs are stored by default in /var/opt/supermarket/data. You can change this to use Amazon S3 (recommended) or an S3-compatible service. If those are not an option you can symlink this directory to shared storage (e.g. NFS) although this has not been fully tested against race conditions.
• (Optional) Caching Service: Supermarket uses Redis as its caching service. You can safely run one Redis instance per Supermarket app server, or you can choose to run a Redis 2.8+ server or HA pair.

Troubleshooting & FAQ

Incorrect redirect URL

The redirect URL specified in oc-id MUST match the hostname of the Supermarket server. Also, you must get the URI correct (/auth/chef_oauth2/callback). If these are not true, you will recieve an error message like:

1

The redirect uri included is not valid.

Supermarket server cannot reach oc-id, throws 500 error during login

The Supermarket server must be able to reach (via https) the specified chef_server_url - it does this during OAuth2 negotation. The most common problems are name resolution and firewall rules.

Where can I find the code to Supermarket?

• Supermarket the rails application is located here
  • All Supermarket issues should be reported there
• The code which builds Supermarket into an Omnibus package is located here
  • The cookbook that is run when during supermarket-ctl reconfigure is located within this repo
  • You can build your own Omnibus packages by following the instructions in the README.md

How do I enable rails application debug logging?

There is a known issue with the Supermarket omnibus package that rails messages are not logged. To fix that requires a manual change at the moment. On your supermarket server, edit this file: /opt/supermarket/embedded/service/supermarket/config/environments/production.rb, change line 46 (config.log_level = :warn) to look like:

1
2
3

  config.logger = Logger.new('/var/log/supermarket/rails/rails.log')
  config.logger.level = 'DEBUG'
  config.log_level = :debug

Then restart the rails service by running supermarket-ctl restart rails

How does this OAuth2 stuff work anyway?

Here’s a simplified description of OAuth2: https://aaronparecki.com/articles/2012/07/29/1/oauth2-simplified

1. When you visit supermarket at https://supermarkethttps://supermarket/ and click login, that login redirects you to https://chef-server/oc-id
2. https://chef-server/oc-id then redirects you back to https://supermarket/auth/endpoint once you are confirmed as authed
3. Supermarket talks to chef-server/oc-id to verify the token it just received by making an https call to the chef server

Contacting packagecloud fails if I’m behind a proxy

No problem! Add the following to your .kitchenl.local.yml file:

1
2
3
4
5
6

---
provisioner:
  name: chef_zero
  solo_rb:
    http_proxy: http://192.168.1.1
    https_proxy: http://192.168.2.2

Test kitchen is slow because it has to download/install the Chef Omnibus client package every time

Here’s a few tips to speed it up:

1. Tell test-kitchen to cache the Omnibus installer (put this in your .kitchen.local.yml file):

   1 2 3 4

   --- provisioner: name: chef_zero chef_omnibus_install_options: -d /tmp/vagrant-cache/vagrant_omnibus

2. Cache yum repos like packagecloud using the vagrant-cachier plugin. First run vagrant plugin install vagrant-cachier, then create a $VAGRANT_HOME/Vagrantfile that looks like so:

   1 2 3 4 5 6 7 8 9 10

   Vagrant.configure("2") do |config| config.vm.box = 'some-box' if Vagrant.has_plugin?("vagrant-cachier") config.cache.scope = :box config.cache.enable :chef config.cache.enable :apt config.cache.enable :yum config.cache.enable :gem end end

Source code at: https://github.com/irvingpop/rhcs7

I created a Vagrant-based configuration that brings up two CentOS 7 nodes that share a cluster disk with CLVM. Based on: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Administration/ch-startup-HAAA.html with clarification from: http://www.davidvossel.com/wiki/index.php?title=HA_LVM

Ultimate goal: Find a way to couple Redhat clustering with Chef Server to create a more robust HA stack.

Notes

• Redhat clustering is HARD and not for the faint of heart. It’s not nearly as well documented as it should be, there’s quite a few closely interconnected services and you should really take time out to understand it on its own before combining it with anything else.
  • Redhat clustering in EL5 and EL6 was pretty awful to operate and hard to understand. It has been majorly redone in RHEL7 and is now considerably simpler to get going.
• This document and repo focus on the shared storage use case, where two nodes have access to the same block device but only one node can actively mount and write to it at a time. The simultaneous access use cases (GFS, OCFS, etc) are not covered here.
• Initially I tried to use Virtualbox “shareable” block devices but this didn’t work because it doesn’t support the SCSI SPC-3 feature set. This is important because the only applicable fence agent I could use is fence_scsi, which uses SCSI SPC-3 persistent reservations ( https://kb.netapp.com/support/index?page=content&id=3012956 )
• I switched to using the Linux iSCSI service on a separate node to get SPC-3. In production you shouldn’t use iSCSI. It will burn your beans, melt your ice cream, and in general disappoint you continuously. Yes, Fibre Channel is expensive and iSCSI is cheap, but over a decade of storage management experience has taught me that there’s a good reason for that.

There’s two ways to do HA-LVM: * Exclusive Activation via LVM volume_list filtering: This basically tells LVM not to auto-activate the shared storage, and then uses a tagging system (plus trust in pacemaker) to control exclusive access * Exclusive Activation via clvmd: This uses the clvmd service (plus a distributed lock manager) to control access to LVM volume groups which are specially tagged as “clustered” * Irving’s thoughts: clvmd would appear to be the safer way to go because it protects against “rogue” nodes, but in practice clvmd is really freakin hard to figure out and troubleshoot. Also, the SCSI SPC-3 persistent reservations provide an additional layer of safety. So I recommend Option 1, the volume_list filtering approach as it gave me far less heartburn in testing.

Initial setup

1. Initial configuration of the cluster machines, backend0 and backend1:

   1	vagrant up

2. iSCSI server setup from: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/ch25.html#osm-target-setup

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

   # install yum install -y targetcli # start and enable the service systemctl start target && systemctl enable target # map the backstore to a physical unformatted block device targetcli /backstores/block create name=block_backend dev=/dev/sdb # create an iSCSI target targetcli /iscsi create iqn.2006-04.com.iscsi-is-awesome:1 # Created target iqn.2006-04.com.iscsi-is-awesome:1 # Created TPG 1. # create the iSCSI portal (IP listener) targetcli /iscsi/iqn.2006-04.com.iscsi-is-awesome:1/tpg1/portals/ create # Using default IP port 3260 # Binding to INADDR_ANY (0.0.0.0) # Created network portal 0.0.0.0:3260. # Map the iSCSI target to the backstore targetcli /iscsi/iqn.2006-04.com.iscsi-is-awesome:1/tpg1/luns/ create /backstores/block/block_backend # Created LUN 0. # crazy wild-west no-ACL mode, because this is a PoC :) targetcli /iscsi/iqn.2006-04.com.iscsi-is-awesome:1/tpg1/ set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 # Parameter demo_mode_write_protect is now '0'. # Parameter authentication is now '0'. # Parameter generate_node_acls is now '1'. # Parameter cache_dynamic_acls is now '1'. # sit back and marvel at your iSCSIs targetcli ls # o- / ........................................ [...] # o- backstores ............................. [...] # | o- block ................. [Storage Objects: 1] # | | o- block_backend [/dev/sdb (1.0GiB) write-thru activated] # | o- fileio ................ [Storage Objects: 0] # | o- pscsi ................. [Storage Objects: 0] # | o- ramdisk ............... [Storage Objects: 0] # o- iscsi ........................... [Targets: 1] # | o- iqn.2006-04.com.iscsi-is-awesome:1 [TPGs: 1] # | o- tpg1 .............. [no-gen-acls, no-auth] # | o- acls ......................... [ACLs: 0] # | o- luns ......................... [LUNs: 1] # | | o- lun0 [block/block_backend (/dev/sdb)] # | o- portals ................... [Portals: 1] # | o- 0.0.0.0:3260 .................... [OK] # o- loopback ........................ [Targets: 0]

3. iSCSI client setup

   1 2 3 4 5 6 7 8 9 10 11

   # install and start the iscsi client service yum install -y iscsi-initiator-utils systemctl start iscsid.service && systemctl enable iscsid.service # connect to the iscsi server iscsiadm -m node -o new -T iqn.2006-04.com.iscsi-is-awesome:1 -p 33.33.33.20:3260 # login iscsiadm -m node -T iqn.2006-04.com.iscsi-is-awesome:1 -p 33.33.33.20:3260 --login # Logging in to [iface: default, target: iqn.2006-04.com.iscsi-is-awesome:1, portal: 33.33.33.20,3260] (multiple) # Login to [iface: default, target: iqn.2006-04.com.iscsi-is-awesome:1, portal: 33.33.33.20,3260] successful.

Base Cluster Setup

1. Run the following commands on both nodes

   1 2 3 4 5 6 7 8 9 10 11 12 13

   # install clustering packages yum -y install pcs fence-agents-all lvm2-cluster # set hacluster user password echo "hacluster" | passwd hacluster --stdin # start clustering services systemctl start pcsd.service systemctl enable pcsd.service # create the mountpoint directory mkdir -p /var/opt/opscode/drbd/data #

2. Run the following commands on the first cluster node only

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

   # authorize cluster pcs cluster auth backend0 backend1 -u hacluster -p hacluster # setup cluster pcs cluster setup --start --name chef-ha backend0 backend1 # enable the cluster and examine status pcs cluster enable --all pcs cluster status # enable SCSI fence mode (uses SPC-3) pcs stonith create scsi fence_scsi devices=/dev/sdb meta provides=unfencing sleep 5 # this might show stopped, try it a few times until it says started pcs stonith show # scsi (stonith:fence_scsi): Started

3. And on the 2nd cluster node:

   1	pcs cluster auth backend0 backend1 -u hacluster -p hacluster

Option 1: Leader Election and LVM Volume failover without CLVM

1. on the first cluster node:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

   # examine cluster status and ensure all resources are Started/Online pcs status # create the LVM PV and VG pvcreate /dev/sdb vgcreate shared_vg /dev/sdb lvcreate -l 80%VG -n ha_lv shared_vg # deactivate the shared_vg, and then reactivate it with an exclusive lock vgchange -an shared_vg # 0 logical volume(s) in volume group "shared_vg" now active lvchange -aey shared_vg # 1 logical volume(s) in volume group "shared_vg" now active # format the volume mkfs.xfs /dev/shared_vg/ha_lv #

2. Update the initramfs device on all your cluster nodes, so that the CLVM volume is never auto-mounted:

   1 2 3 4 5 6 7 8 9

   # Set the lvm configuration to not auto-mount anything but the root "centos" VG # yes, the 783 part means the 783rd line of the file :\ sed -i.bak "783s/.*/ volume_list = [ \"centos\", \"@`hostname -s`\" ]/" /etc/lvm/lvm.conf # update initramfs and reboot dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r) # shutdown, but use vagrant to start them up shutdown -h now #

3. then bring both nodes back up

   1	vagrant up backend0 backend1

4. Add an LVM resource

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

   # create resources for the LVM, filesystem and IP pcs resource create ha_lv ocf:heartbeat:LVM volgrpname=shared_vg exclusive=true --group chef_ha pcs resource create chef_data Filesystem device="/dev/shared_vg/ha_lv" directory="/var/opt/opscode/drbd/data" fstype="xfs" --group chef_ha pcs resource create backend_vip IPaddr2 ip=33.33.33.5 cidr_netmask=24 --group chef_ha # # debugging - only if things are going wrong # run lvs on both nodes to ensure it only says active ("a") on backend0 lvs # on backend0 # LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert # root centos -wi-ao---- 38.48g # swap centos -wi-ao---- 1.03g # ha_lv shared_vg -wi-ao---- 3.20g lvs # on backend1 # LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert # root centos -wi-ao---- 38.48g # swap centos -wi-ao---- 1.03g # ha_lv shared_vg -wi------- 3.20g export OCF_RESKEY_volgrpname=shared_vg OCF_RESKEY_exclusive=true OCF_ROOT=/usr/lib/ocf bash -x /usr/lib/ocf/resource.d/heartbeat/LVM start # or pcs resource debug-start ha_lv pcs resource debug-start chef_data pcs resource debug-start backend_vip

Option 2: Using CLVMD

1. On the first node: Enable clvmd and LVM clustering

   1 2 3 4 5 6 7 8 9 10

   pcs resource create dlm ocf:pacemaker:controld clone on-fail=fence interleave=true ordered=true pcs resource create clvmd ocf:heartbeat:clvm clone on-fail=fence interleave=true ordered=true pcs constraint order start dlm-clone then clvmd-clone pcs constraint colocation add clvmd-clone with dlm-clone # enable LVM clustering with clvmd lvmconf --enable-cluster # stop lvmetad killall lvmetad #

2. Run the following on the second cluster node:

   1 2 3 4 5

   lvmconf --enable-cluster # stop lvmetad killall lvmetad #

3. Back to the first cluster node:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

   # examine cluster status and ensure all resources are Started/Online pcs status # create the LVM PV and VG pvcreate /dev/sdb vgcreate -cy shared_vg /dev/sdb lvcreate -l 80%VG -n ha_lv shared_vg # deactivate the shared_vg, and then reactivate it with an exclusive lock vgchange -an shared_vg # 0 logical volume(s) in volume group "shared_vg" now active vgchange -aey shared_vg # 1 logical volume(s) in volume group "shared_vg" now active # run lvs on both nodes to ensure it only says active ("a") on backend0 lvs # on backend0 # LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert # root centos -wi-ao---- 38.48g # swap centos -wi-ao---- 1.03g # ha_lv shared_vg -wi-a----- 3.20g lvs # on backend1 # LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert # root centos -wi-ao---- 38.48g # swap centos -wi-ao---- 1.03g # ha_lv shared_vg -wi------- 3.20g # format the volume and mount it mkfs.xfs /dev/shared_vg/ha_lv

4. Add an LVM resource (this is where shit gets whacky)

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

   # first unmount and deactivate vgchange -an shared_vg # 0 logical volume(s) in volume group "shared_vg" now active pcs resource create ha_lv ocf:heartbeat:LVM volgrpname=shared_vg exclusive=true --group chef_ha pcs resource create chef_data Filesystem device="/dev/shared_vg/ha_lv" directory="/var/opt/opscode/drbd/data" fstype="xfs" --group chef_ha pcs resource create backend_vip IPaddr2 ip=33.33.33.5 cidr_netmask=24 --group chef_ha # debugging pcs resource debug-start ha_lv pcs resource debug-start chef_data pcs resource debug-start backend_vip export OCF_RESKEY_volgrpname=shared_vg OCF_RESKEY_exclusive=true OCF_ROOT=/usr/lib/ocf bash -x /usr/lib/ocf/resource.d/heartbeat/LVM start

Now you’re done!

Failing over

the pacemaker way

1. bringing down backend0
2. Option 1: shut down backend0 shutdown -h now

3. Option 2: make it a standby via Pacemaker pcs cluster standby backend0

4. failover of resources should be automatic to the secondary node:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

   pcs status # Cluster name: chef-ha # Last updated: Sat Mar 14 21:55:56 2015 # Last change: Sat Mar 14 21:43:21 2015 via cibadmin on backend0 # Stack: corosync # Current DC: backend1 (2) - partition with quorum # Version: 1.1.10-32.el7_0.1-368c726 # 2 Nodes configured # 4 Resources configured # # # Online: [ backend1 ] # OFFLINE: [ backend0 ] # # Full list of resources: # # scsi (stonith:fence_scsi): Started backend1 # Resource Group: chef_ha # ha_lv (ocf::heartbeat:LVM): Started backend1 # chef_data (ocf::heartbeat:Filesystem): Started backend1 # backend_vip (ocf::heartbeat:IPaddr2): Started backend1 # # PCSD Status: # backend0: Offline # backend1: Online # # Daemon Status: # corosync: active/enabled # pacemaker: active/enabled # pcsd: active/enabled

5. Bringing backend0 back up and logging in, it won’t want to run pacemaker

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

   pcs status # Error: cluster is not currently running on this node systemctl start pacemaker # now checking status pcs status # Cluster name: chef-ha # Last updated: Sat Mar 14 22:06:25 2015 # Last change: Sat Mar 14 21:43:21 2015 via cibadmin on backend0 # Stack: corosync # Current DC: backend1 (2) - partition with quorum # Version: 1.1.10-32.el7_0.1-368c726 # 2 Nodes configured # 4 Resources configured # # # Node backend0 (1): pending # Online: [ backend1 ] # # Full list of resources: # # scsi (stonith:fence_scsi): Started backend1 # Resource Group: chef_ha # ha_lv (ocf::heartbeat:LVM): Started backend1 # chef_data (ocf::heartbeat:Filesystem): Started backend1 # backend_vip (ocf::heartbeat:IPaddr2): Started backend1 # # PCSD Status: # backend0: Online # backend1: Online # # Daemon Status: # corosync: active/enabled # pacemaker: active/enabled # pcsd: active/enabled

6. If backend0 is considered standby, you can just unstandby it:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14

   pcs cluster unstandby backend0 pcs status # Cluster name: chef-ha # Last updated: Sat Mar 14 22:11:53 2015 # Last change: Sat Mar 14 22:11:52 2015 via crm_attribute on backend0 # Stack: corosync # Current DC: backend1 (2) - partition with quorum # Version: 1.1.10-32.el7_0.1-368c726 # 2 Nodes configured # 4 Resources configured # # # Online: [ backend0 backend1 ]

7. What happens if you try to mount the disk on the inactive node?

   1 2 3 4 5 6 7 8 9 10

   # LVM will be nice, but won't let you vgchange -aey shared_vg # 0 logical volume(s) in volume group "shared_vg" now active lvchange -aey shared_vg/ha_lv lvs # LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert # root centos -wi-ao---- 38.48g # swap centos -wi-ao---- 1.03g # ha_lv shared_vg -wi------- 816.00m

the CLVM way

1. On the active node

   1 2 3 4 5 6 7 8 9

   [root@backend0 ~]# umount /var/opt/opscode/drbd/data [root@backend0 ~]# lvchange -an shared_vg/ha_lv # ensure the LV isn't active [root@backend0 ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert root centos -wi-ao---- 38.48g swap centos -wi-ao---- 1.03g ha_lv shared_vg -wi------- 3.20g

2. on the standby node

   1 2 3 4 5 6 7

   [root@backend1 ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert root centos -wi-ao---- 38.48g swap centos -wi-ao---- 1.03g ha_lv shared_vg -wi------- 3.20g [root@backend1 ~]# lvchange -aey shared_vg/ha_lv [root@backend1 ~]# mount /dev/shared_vg/ha_lv /var/opt/opscode/drbd/data

Forcing yourself to go standby

TBD

Forcing the other node to go standby (STONITH)

TBD

Troubleshooting

Do SCSI SPC-3 persistent reservations work on my device?

good:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

# exampling talking to the Linux iSCSI service:
sg_persist --in --report-capabilities -v /dev/sdb
#    inquiry cdb: 12 00 00 00 24 00
#  LIO-ORG   block_backend     4.0
#  Peripheral device type: disk
#    Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00
# Report capabilities response:
#  Compatible Reservation Handling(CRH): 1
#  Specify Initiator Ports Capable(SIP_C): 1
#  All Target Ports Capable(ATP_C): 1
#  Persist Through Power Loss Capable(PTPL_C): 1
#  Type Mask Valid(TMV): 1
#  Allow Commands: 1
#  Persist Through Power Loss Active(PTPL_A): 0
#    Support indicated in Type mask:
#      Write Exclusive, all registrants: 1
#      Exclusive Access, registrants only: 1
#      Write Exclusive, registrants only: 1
#      Exclusive Access: 1
#      Write Exclusive: 1
#      Exclusive Access, all registrants: 1

bad:

1
2
3
4
5
6
7
8
9
10

# example using Virtualbox SCSI/SAS/etc disks which are not SPC-compliant:
sg_persist --in --report-capabilities -v /dev/sdb
#    inquiry cdb: 12 00 00 00 24 00
#  VBOX      HARDDISK          1.0
#  Peripheral device type: disk
#    Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00
# persistent reservation in:  Fixed format, current;  Sense key: Illegal Request
#  Additional sense: Invalid command operation code
#   Info fld=0x0 [0]
# PR in (Report capabilities): command not supported

TBD

Daisy, Daisy, etc etc.

def foo
  puts 'wat'
end

1
2
3

def blah
  puts 'woohoo'
end

github gist: