Tags:
create new tag
view all tags

Puppet

Overview of puppet in phoenix

This wiki page is an overview of how Puppet it set up on Phoenix. It assumes you have a working understanding of Puppet and covers aspects that are specific to the setup within our environment. It also incorporates some git basics.

  • Puppet masters: phoenix1
  • PuppetDB servers: phoenix1

  • TODO Run puppet behind apache/ nginx rather than the default ruby webrick server.
  • TODO Setup phoenix2 as another puppet master and load balance.
  • TODO Due to puppet only allowing a singe CA we could still have a single point of failure.
    • Could possibly use HA proxy to direct SSL traffic and document a manual fail-over. Wont be able to do this automatically as if the CA is unavailable puppet wont run.
    • Could request generic DNS name such as phoenix.lcg.cscs.ch and have both masters mount the CA dir via NFS.
  • TODO Setup phoenix2 as another puppetdb server and load balance. Not really a load issue but more for HA.
    • Requires postgres replication and HA proxy magic, likely has SSL complications but should be workable.
  • TODO Setup git post receive hooks on git remotes to ensure bad code is not allowed to be committed.
    • Pass a style guide check (puppet-lint, omit checks such as 80 characters per line)
    • Pass a syntax check (puppet parser validate)
    • Ensure there is no white space

CSR Attributes

In puppet 3.4 a new feature was added that allowed for custom attributes to be embedded within the CSR (certificate signing request). We can use this is implement a more secure auto signing policy. The following shows how this could be set up, obviously puppet manages this.

Fist the client must have the /etc/puppet/csr_attributes.yaml file present before sending the request. Below is an example of the contents. As you can see if follows the YAML format, under the custom_attributes section we have defined an OID (Object Identifier) and assigned the value foo.

custom_attributes:
  1.2.840.113549.1.9.7: foo

On the server we can see upon examining the CSR recived from the client on the puppet master we have our custom attribute embedded in the request. Note how "1.2.840.113549.1.9.7" is interpreted as "challengePassword".

sudo openssl req -noout -text -in  /var/lib/puppet/ssl/ca/requests/wn45.lcg.cscs.ch.pem | grep Attributes -A 1
        Attributes:
            challengePassword        :foo

Our puppet master to match the attribute need to be able to check the attribute. This will simply be an executable that returns 0 for successes or any other valure for failure. Puppet executes this script and if it receives 0 as the return code the certificate is signed.

Firstly we need to tell puppet were our autosign policy is located. By telling puppet this information it also enables policy autosign.

vi /etc/puppet/puppet.conf

  autosign = /etc/puppet/autosign.sh

In this instance we will use a simple bash script to check the custom atribute is what we are expecting. The autosign script it passed the hostname of the client as an argument and the CSR in PEM format as stdin see http://docs.puppetlabs.com/puppet/latest/reference/ssl_autosign.html#policy-executable-api

vi /etc/puppet/autosign.sh

#!/bin/bash

HOST=$1
CUSTOM_ATTR=$(openssl req -noout -text -in "/var/lib/puppet/ssl/ca/requests/$HOST.pem" | grep "challengePassword" | awk -F ":" '{print$2}')

if [ [ $CUSTOM_ATTR == "foo" ] ]
then
  exit 0
else
  exit 1
fi

We are not limited to a simple pre-shared key, we could for instance use the MAC address or some other fact that is unique to each node as our custom attribute. Then on our master we could check against a white list perhaps referncing an ENC or inventory system.

Hiera

ENC

We use Hiera as an ENC, to do so we need the following. This allows hiera to lookup classes.

cat /etc/puppet/manifests/site.pp 
hiera_include('classes')

This allows us to declare classes in our hiera data for example our common.yaml file defines the classes all nodes will have applied.

head /etc/puppet/hieradata/common.yaml
---
classes:
  - rpmrepos::epel

  - puppet::install
  - puppet::agent
  - puppet::autosign
  - puppet::service

Hiera lookups for paramaterised classes are perfomed as normal

grep ntp /etc/puppet/hieradata/common.yaml

ntp::servers: [ "time1.cscs.ch", "time2.cscs.ch" ]
ntp::restrict: [ '127.0.0.1' ]

Hieradata

Hieradata is managed by the hiera module. This essentially ensures that the hieradata has the latest revision from the master branch in the phoenix hierdata git repo. We do this using the vcsrepo type.

R10K

We use R10K to deploy our modules that are either hosted on an internal git server or directly from the forge. It is a tool that uses the puppet file format, it is similar to librarian-puppet but allows us to easy deploy environments as well.

Essentially the R10K reads a puppet file from a git repo and uses that to deploy modules. The clever thing is that it uses git branches on this repo to define environments. For example the contents of the file under the production branch would be the modules installed to our production environment. Likewise a branch named test would be our test environment. This allows us to create dynamic environments without having to have all of our puppet code under a single git tree.

Config

Again R10K is installed and managed by puppet although deploys are performed manually.

The configuration for R10K is very simple.

cat /etc/r10k.yaml 
:cachedir: '/var/cache/r10k'
:sources:
  :phoenix:
    remote: 'git.cscs.ch:/git/puppet/modules/phoenix.git'
    basedir: '/etc/puppet/environments'

The key points for the file are the remote and basedir settings. The remote refers to the location of the git repo containing the puppet file and the basedir is the top level for our puppet environments (our puppet.conf must reflect this). For example once deployed we may have something that looks like the following

ls -l /etc/puppet/environments
total 12
drwxrwxr-x 3 puppet cscs 4096 20. Feb 15:33 master
drwxrwxr-x 4 puppet cscs 4096  7. Mär 10:33 preproduction
drwxrwxr-x 4 puppet cscs 4096 17. Mär 15:30 production

Install a module from the forge

The syntax to use in the puppet file for installing a module from the forge is as follows

# Use the public puppet forge
forge 'http://forge.puppetlabs.com'

# Install puppetlabs/concat from the forge
mod 'puppetlabs/concat'

Install a module from git

To install a module from git we can simply do the following

mod 'ntp',
  :git => 'git.cscs.ch:/git/puppet/modules/ntp.git'

If we want to ensure a specific branch or tag is checked out we can use the "ref" attribute. In the following example we are tracking a branch named phoenix. Without a refernce master is simply tracked.

mod 'dhcp',
  :git => 'git.cscs.ch:/git/puppet/modules/dhcp.git',
  :ref => 'phoenix'

Example usage

Deploy production

If you have been working on some changes to a module(s) and are ready to deploy to production ensure the puppet file in the production branch is up to date and enter the following

sudo r10k deploy environment production -pvt

I always like to use the -v and -t options for verbose and trace respectively. The -p ensures the puppet files are in sync with what is stored in the remote.

Deploy a test environment

If we wanted to create a test environment we would do the following. Lets say we had been working on the puppet module to run puppet behind apache, we have made our changes on a branch of the puppet module named "apache" and we want to deploy it to machines only in the testing environment. We could make a more substantial change which involves multiple module changes and checkout a branch called EMI4 for example.

# Clone and checkout testing

git clone git.cscs.ch:/git/puppet/modules/phoenix.git
cd phoenix
git checkout -b testing

# Edit the file

vim Puppetfile

  mod 'puppet',
  :git => 'git.cscs.ch:/git/puppet/modules/puppet.git'
  :ref => 'apache'

# Commit and push

git commt -a
git push origin testing

# Deploy
sudo r10k deploy environment testing -pvt

How to fork a remote module

Sometimes it is necessary to clone a module from another repository (github for example).

To do this, simply follow these commands:

git clone https://github.com/cernops/puppet-glexecwn.git
git remote add origin miguelgi@git.cscs.ch:/git/puppet/modules/puppet-glexecwn.git
git branch --set-upstream master cscs/master

-- GeorgeBrown - 2014-03-18

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2014-03-20 - MiguelGila
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback