Configuration Management with Puppet, MCollective and Chocolatey on Windows

[VedranFX]

Configuration Management with Puppet, MCollective and Chocolatey on Windows

1. Puppet introduction
   1.1 Puppet server installation
   1.2 Puppet agent installation
2. MCollective
   2.1 Deployment
   2.2 Installing middleware
   2.3 Installing mcollective
   2.4 Configuring MCollective on Admin machines
3. Workflow
   3.1 Custom facts
   3.2 Maya license type
   3.3 Chocolatey packages
   3.3 Role
   3.4 Hiera
   3.5 R10K
   3.6 Roles and Profiles
   3.7 Modified open source components
4. Chocolatey
   4.1 Chocolatey defaults
5. Development environment
   5.1 Quick start
   5.2 Working with Development environment
   5.3 Advanced development

Infrastructure automation tools, such as Puppet, MCollective and Chocolatey, are used to quickly set up new machines or update their configuration (e.g. update software, security policies or operating system settings).

Puppet introduction

Puppet is a Configuration Management (CM) tool that uses the Domain Specific Language (DSL) to specify the desired state of the system and its resources. It has important characteristic called idempotency, which means that it can be executed multiple times and the result will always be the same.

Fig 1.0, Puppet workflow cycle

The desired state is described in a puppet manifest file. This file can then be applied by puppet agent, a puppet component that runs in all nodes.

The following is an example of a puppet manifest that:

• Creates a user
• Installs a package, only if the user was created first

user { 'admin':
  ensure   => present,
  password => "supersecret",
}

package { 'vzpython':
  ensure   => present,
  provider => 'chocolatey',
  require  => User['admin']
}

In the previous example, we have two types of resources: user and package. Each of them have specific parameters.
By using the require parameter on the package resource, we are creating a dependency on the user one. This means that in order to execute the package resource, the user one has to be executed successfully previously.

Puppet can be used in two different ways, with puppet server and standalone.

When using puppet server, all puppet manifests are stored on central server, and server sends catalog to puppet agent. To create catalog, puppet server compiles manifests files into list of resources and their desired state.

The following steps are typical puppet workflow:

1. Puppet agent runs and dowloads all the necessary content from the server (e.g. custom facts and plugins)
2. Puppet agent sends facts about itself to server, and requests catalog
3. Puppet server uses agent facts, and logic from puppet manifests to compile a catalog and sends it to agent
4. Puppet agent applies the catalog, by synchronizing all managed resources to desired state
5. Puppet agent sends the report to puppet server

This basic workflow can be visualised as the following illustration:

Puppet can also work without a server, by using the command puppet apply somemanifest.pp

Puppet server installation

For now, Vetor Zero’s puppet infrastructure is running in docker containers.

A docker compose file was created, which runs multiple components:

• Puppet server
• PuppetDB
• PostgreSQL
• Puppet Board WebUI

The repository which contains the configuration files is: https://github.com/vetorzero/puppet-env

To start using puppet server:

1. Download the docker-compose.yml file
2. Run docker-compose up on a docker host.

Puppet agent installation

We are using the latest enterprise supported version of puppet, which is currently at version 4.10. We chose this version due to its stability.

The installation of puppet agent (in Windows) is very simple, and it only requires 2 steps:

1. Specifying the Fully Qualified Domain Name (FQDN) of the puppet server [REQUIRED]. The default value is puppet.

2. Then, test puppet agent’s connectivity by executing puppet run using simulation mode.
   2.1 Open a Command Prompt with Puppet as an Admnistrator which you can find in your start menu

   

   2.2 Type puppet agent --test --noop

   

   As you can see in the output, the first thing that happens is that puppet agent will send a certificate request to puppet server, which will then be signed, and accepted. Afterwards, puppet agent will download all plugins, modules, custom facts, etc, and a simulation of a puppet run will be displayed.

Note: We have enabled automatic signing of certificates, since we have controlled environment. Puppet labs recommends to manually sign
certificates on the server. By running puppet cert list we can see any pending requests, and then sign them by executing puppet cert sign name-of-the-host.
For more information refer to Puppet documentation on autosign.conf.

Marionette Collective (MCollective)

MCollective is a framework for building remote execution tools and it is integrated with CM tools such as Puppet. It can run commands in parallel on thousands of machines. To achieve this, it does not connect to each machine, but instead uses a message queue, known as middleware, for communication with remote nodes.
It uses a model of clients and servers. A client sends a message to the message queue, and the servers who are subscribed to a specific message receieve it, and act upon it through their agents, which execute actions.

It is possible to use authentication and authorization to limit specific machines and users to specific actions.
In the case of Vetor Zero, we are using MCollective as a complement to puppet in order to trigger puppet runs. Executed by demand of a user, a boot script, or a remote admin. All this without requiring the local user to have administrative privileges on the host operating system.

Deployment

MCollective is deployed using great Choria mcollective deployment tool.

Choria configures servers, clients, middleware and it uses strong SSL certificates from Puppet 4 CA (Certificate Authority). So in order to use Choria, we need to have
Puppet 4 server, and we also need to have Puppet installed on client machine, since it re-uses puppet certificate to communicate with middleware.

In order to start using choria, we need to install puppet module puppet module install choria/mcollective_choria, which will install all the required dependencies. Choria also packages mcollective plugins (e.g agents, security) as special puppet modules, so only system package installed is mcollective.

Installing middleware

Middleware used with Choria is NATS, which is very simple to configure, a single binary and configuration file.
NATS, is currently running in the puppetserver container. Execute the following commands for installation

To install nats on puppetserver we need to have nats puppet module. To install this puppet module run:
puppet module install choria/nats

Then, to apply nats configuration for use with choria:
puppet apply -e 'include nats' --modulepath /etc/puppetlabs/code/environments/production

Finally, execute middleware using generated config file:
/usr/sbin/gnatsd --config /etc/gnatsd/gnatsd.cfg

Installing MCollective

Deploying mcollective with Choria is really easy. We just need to include mcollective puppet module, and specify configuration settings using hiera.
There is existing chocolatey package for mcollective, which works transparently with mcollective puppet module. This package only contains mcollective core files.

Snippet from from vetorzero site.pp manifest:

Package {
  provider => 'chocolatey',
  source   => $repo_source,
}

include ::mcollective
include base_windows

Class['::mcollecive'] -> Class['base_windows']

Using hiera configuration data, choria automatically configures:

• Server configuration file - server.cfg
• Client configuration files - client.cfg
• Choria configuration
• Action policy authorization policies
• Auditing

NOTE: Very important configuration setting is to set mcollective::client: true, since choria otherwise will not install neccessary client plugins

Vetor Zero choria configuration is defined in the following file: https://github.com/vetorzero/puppet-control-repo/blob/dev/hieradata/common.yaml

By default, each mcollective client is limited executing commands on it’s own host for security purposes.

Configuring MCollective on Admin machines

In order for an administrator to execute mcollective commands on multiple hosts, some special preparation is required.
Choria configures mcollective to use SSL certificates from existing puppet SSL infrastructure, so by default it uses a certname of hostname which is registered with puppet server.

Since hostname certname is only allowed specific commands we created a special admin certname, and defined authorization policies to allow any command on any host, only by this special admin certificate.

This is very secure, since malicious user would need to have certificate files, in order to control mcollective.

Steps to enable admin mode:

1. Copy private_keys\admin.pem and place it in %ProgramData%\PuppetLabs\puppet\etc\ssl\private_keys\
2. Copy certs\admin.pem and place it in %ProgramData%\PuppetLabs\puppet\etc\ssl\certs\
3. Open command prompt and run set MCOLLECTIVE_CERTNAME=admin

Afterwards, running mcollective commands will be run with admin privileges

NOTE: Take special CAUTION running mcollective commands using admin certificate without any kind of filtering. Any mcollective command and action will be executed on all hosts! In order to limit execution on specific hosts, use something like mco shell puppet -I /^HOSTNAME$/

Workflow

We have established Puppet workflow that uses following components:

• Custom facts
• Hiera
• R10K
• Roles and Profiles

There is also an vagrant development environment, which is using exact same puppet code for development and testing.

Vetor Zero puppet environment, also uses some modified open source components:

• MCollective shell agent
• Choria

Custom facts

A number of custom facts were written in order to faciliate migration from current software distribution in pipeline to chocolatey and puppet.

Currently we have written:

• Maya license type fact
• Chocolatey packages fact
• Role fact

Maya license type

There is a specific requirement for Maya 2017 deployment packages, since we have multiple types of licenses available, Standalone, Network, and Disconnected.

In order to automate deployment, 3 chocolatey packages were created, and hiera uses maya_license_type fact to determine package name to install.
This custom fact uses a file currently at //nibbler2/apps/MayaDeploys/Maya2017/standalone.txt and compares currently logged-in user with the users found in file.

if the logged-in user is also in the file, fact returns “Standalone”, otherwise “Disconnected”.
There is specific case for renderfarm machines, where this fact compares if the role of the machine is renderfarm, then it sets the fact to "Network"

Chocolatey packages

In order to detect installed package, packages fact queries chocolatey for both local and remote package. This information is used for vzTrayService, in order to display the status of the host to the user (e.g if some packages are available but not installed), and it is also used by TSAdmin for the same purpose.
This structured fact return 3 keys, local_packages, difference and size.

Difference is displaying all the packages that are available remotely, but not locally, or packages with the same name but different version locally.

NOTE: When remote version is some version, and local “0.0.0”, this means that package is not installed locally

Example json output:

{ 
  "packages" : {
    "local_packages": {
      "vzpython": "1.1.5",
      "maya2017": "1.0.0",
      "vzcmd": "1.0.1"
    },
    "difference": {
      "vzpython": {
        "local": "1.0.0",
        "remote": "1.1.5"
      },
    "maya2018": {
      "local": "0.0.0",
      "remote": "1.0.0"
      }
    },
    "size": 2
  }
}

Role

Role is a simple fact and it exists in order to be able to easily specify a type of the host, such as renderfarm, workstation or even infrastructure server.
By using hiera, we can then include specific profiles (as in Roles and Profiles design pattern), depending on the fact result.

Fact works like this:
If current hostname is logged-in as renderfarm then it is set to renderfarm Otherwise it’s set to workstation.

Hiera

Hiera is a hiearchical key-value store that is integrated with puppet and allows us to simplify our puppet code.
It allows us to separate data from the code, and it opens opportunities for non-technical personnel to understand the configuration of hosts without looking at the puppet code.

It has few interesting features such as:

• Automatic parameter lookup
• Hierarchical search
• External key-value store

1. Automatic parameter lookup

This feature allows declaration of classes just by using include statement, without specifying any parameter.

When compiling catalog, puppet first looks into hiera, and if it finds a key that matches class name and parameter, it uses it.

For example if we have defined an puppet class:

class base::package (
  String $package_name = undef,
  Boolean $system_tools = False
) {

  package { $package_name:
    ensure => 'latest',
  }
}

And we have a key in hiera data file that specifies class parameter:

base::package::package_name: 'vzcmd'
base::package::system_tools: true

Then we can declare the class just by saying include base::package and it will read the parameters from hiera. This means that
different hosts can recieve different parameters, and all this without any conditional logic.

2. Hierarchical search

In order to work with Hiera we need to define search hierarchy for keys, where specific yaml can be found.

Example hierarchy with hiera:

- role/%{facts.role}.yaml
- os/%{facts.os.family}.yaml
- common.yaml

In ths example we have a file called common.yaml, in which we put class parameters, and settings, for all hosts, but then we can override
some settings in in the file os/windows.yaml (os is a folder in this case), so we can specify specific parameters only for windows hosts.

By using such hierarchical search, we can minimize use of conditional logic in the puppet code.

3. External key-value store

Hiera can also be used as an external database, where we can query for any key using puppet code.

if we have a key called custom_variable in any hiera data file, then we can get the value of the key by using:

$variable = hiera('custom_variable')

Again, same rules of hierarchical search applies to direct queries as well.

R10K

R10K is deployment tool that turns branches from git repositories into puppet directory environments. With puppet environments we can safely separate production from development and testing, just by having multiple git branches.

A source repository, known as control repository, contains Puppetfile, R10K can automate
deployment of all puppet modules and custom code.

Puppetfile contains a list of all modules and versions, from both private modules and public modules on puppet forge.

In order to update puppet environment we just need to run:

r10k deploy --environment $environment -pv

If we create a new branch called testing, and then deploy it to puppet server, any puppet agent can be called by specifying this environment, in order to develop and test changes in isolation.

by running puppet agent -t --environment testing we will only apply the changes on that node, and would not affect production.

Roles and profiles

Roles and profiles is best practice design pattern recommended by puppet labs for easier management and organization of puppet classes.
Both role and profile is a simple puppet class, that includes other component classes.

The idea is that profile represents a specific technology, and role is natural role that uses other profiles.

if we have a class profile::webserver it might include other component classes, that deal with web server components:

class profile::webserver {
  include apache
  include mysql
  include base::security
}

Then we can create another class, role::intranet that includes those profiles:

class role::intranet {
  include profile::webserver
}

NOTE: Roles and profiles design pattern helps when classifying hosts, and it also defends us when we have multiple identical resources declared in different classes, we avoid duplicate declaration in puppet catalog which results in compilation failure.

Modified opensource components

1. Choria

Choria is MCollective deployment tool. In order to make it work in vetorzero environment, a small modification was required.
By default, choria does not allow executing as root, and choria’s code checks for Uid 0 (root) which does not exist on windows.

A small fix was added to ignore checking for root when on windows.

2. MCollective shell agent

Shell agent is MCollective plugin that allows us to execute any shell command over mcollective.
Leaving shell agent as is, would allow anyone executing MCollective commands locally as an admin, and in order to secure mcollective, we modified shell plugin so it has special action called puppet, which when called, just calls puppet command.

By doing this, we can use another MCollective plugin Authorization Policy, to allow executing only puppet action on shell plugin. Every other action including the most dangerous run is forbidden.

NOTE: By default every action for every plugin is forbidden unless authorized by authorization policy, defined in puppet hiera configuration

Chocolatey

Chocolatey is installed using puppet base module, where chocolatey class is already declared.

In order to configure chocolatey itself, we are using chocolatey custom types chocolateyconfig and chocolateyfeature.
These custom resources are using parameters passed to puppet as Hash, that are stored in hiera.

For example, if we want to enable feature for allowing global confirmation, in Puppet DSL:

chocolateyfeature { 'enableglobalconfirmation':
  ensure => enabled,
}

Rather then needing to specify resource parameters, we are using hiera data, and any feature or config is applied by puppet.

base_windows::choco_options:
  feature:
    enableglobalconfirmation: 'enabled'
    stoponfirstpackagefailure: 'enabled'
  config:
    commandExecutionTimeoutSeconds: '3600'

This approach has benefit of not needing to edit any puppet manifest in order to include a new choco configuration.

Vetor Zero chocolatey defaults

Some chocolatey settings were modified to better suit vetorzero environment. The following is short explanation.

commandExecutionTimeoutSeconds

Default	Vetorzero
2700	3600

Reason for change: We have some very big packages, or packages that take very long time to install

logEnvironmentValues

Default	Vetorzero
False	True

Reason for change: It’s useful to have environment variables used during package installation

stopOnFirstPackageFailure

Reason for change: When installation of package dependency fails, we want to stop package installation and fail early.

NOTE: This feature does not mean that when package installation fails, others won’t start. It just means that single package will fail, if it
can’t satisfy it’s dependency. Since we are using puppet, other packages will be installed as normal. Puppet package dependency is between
sys and pipe types of packages, where all pipeline packages will be skipped, if any of the system package installation is not sucessfull.
This is not related to chocolatey setting.

useRememberedArgumentsForUpgrades

Default	Vetorzero
False	True

Reason for change: TODO

allowGlobalConfirmation

Default	Vetorzero
False	True

Reason for change: Accept confirmation prompts that could appear during package installation, this is neccessary for running chocolatey unattended.

allowEmptyChecksums

Default	Vetorzero
False	True

Reason for change: All packages are going to be created internally, so checksums are not of big concern. Even packages downloaded from external sources
will be repackaged. Without this option some security confirmation prompts might appear.

failOnStandardError (TEST)

Default	Vetorzero
False	True

Reason for change: We would like for package installation to fail, even if installation succeeds but there were some errors.

Development environment

Puppet development environment is available as a vagrant image at Github.
By using the same puppet code that is in central repository it allows any developer to use puppet for testing and development purposes.

Requirements:

In order to use this, you need to first have Vagrant and Virtualbox installed. Vagrant is available at http://www.vagrantup.com and Virtualbox at http://www.virtualbox.org
Make sure to restart your machine after installation.

Quick start

1. Add vagrant box with windows 10.

Open your shell and type vagrant box add --name windows_10 \\srv-replicant\_TEMP_\blabla\windows10.box

C:\Users\vedran.sinobad\Vagrant\Windows_10
λ vagrant box add --name windows_10 \\srv-replicant\_TEMP_\Vedran\VMS\windows10.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'windows_10' (v0) for provider:
    box: Unpacking necessary files from: file:////srv-replicant/_TEMP_/Vedran/VMS/windows10.box
    box: Progress: 100% (Rate: 48.1M/s, Estimated time remaining: --:--:--)
==> box: Successfully added box 'windows_10' (v0) for 'virtualbox'!

If you get an error, saying that it cannot find vagrant command, please check that vagrant is in your environment PATH variable.
By restarting the host it should automatically add it there, if not add C:\HashiCorp\Vagrant\bin to PATH.

2. Clone puppet-dev-environment repository and start vagrant

git clone git@github.com:vetorzero/puppet-dev-environment
cd puppet-dev-environment/Windows10
vagrant up

λ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'windows_10'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: Windows 10 Puppet Environment
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 3389 (guest) => 3389 (host) (adapter 1)
    default: 5985 (guest) => 55985 (host) (adapter 1)
    default: 5986 (guest) => 55986 (host) (adapter 1)
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
...

3. Enter into powershell session when virtual instance is running

vagrant powershell

Alternatively you can login to your instance using Virtualbox GUI that opened while machine was starting, Login and password is vagrant.

NOTE: It seems there is an issue of accessing vagrant virtual machine using vagrant powershell from Windows 7. Disabling the firewall in the guest
operating system seems to work. This does not happen on Windows 10 host, and it works normally with enabled firewall.

Great thing about vagrant is that it automatically configures shared folder that is mounted at C:\vagrant and this allows us to put our code there
and edit it live on development machine, while executing it on vagrant instance.

In our case shared folder is folder where you cloned puppet-dev-environment.

4. Download all puppet code into vagrant shared folder

On your host operating system, navigate to folder where you cloned puppet-dev-environment and run:

git clone --branch dev --recursive git@github.com:vetorzero/puppet-control-repo
git submodule update --remote

We need to use --recursive flag when cloning since, puppet-control-repo contains submodules. Second command will checkout the latest commit in each submodules, and update the state of the repositories.

5. Extract public modules from modules.zip file into puppet-control/modules folder and run puppet.

C:\vagrant\puppet-dev-environment\run_puppet.bat

This should start puppet in local mode, and this script simply configures all the paths to modules, manifests and hiera in shared vagrant folder.

After the first puppet run, mcollective will be installed and you can use mcollective to run puppet even as non-admin user, by running mco shell puppet --tail.
This is default behaviour at vetorzero.

Working with Development environment

If you finished quick start section, you should have everything ready to start developing puppet modules.
run_puppet.bat batch script is really single line with many parameters specified on command line. In order to simulate running the same environment
that is available on Puppet server, we need to specify

• Modules path
• Hiera configuration path
• Entry point manifest (by default this is site.pp)

For example, if your code is in default folder such as C:\Vagrant\puppet-control, then specifying the following command allows us to run puppet exactly the same as if we ran
puppet agent -t

puppet apply --modulepath=C:\Vagrant\puppet-control\modules;C:\Vagrant\puppet-control\vetorzero --hiera_config=C:\Vagrant\puppet-control\hiera.yml C:\Vagrant\puppet-control\manifests\site.pp

Now, it would be best time to configure a project in your text editor or IDE, which points to where your vagrant folder is. By editing this code on your host machine,
you can then run it immediately in test environment. This allows for rapid development and testing.

Before any work done in the test environment, i recommend to shutdown your vagrant instance and make a snapshot with the following commands:

vagrant halt
vagrant snapshot save clean-start

The last command will create a virtual machine snapshot called clean-start in this case, which allows you to rollback the state of the vagrant instance.
In order to restore the machine to the clean state you would use:

vagrant snapshot restore clean-start

NOTE: By having a virtual isolated environment, you could also run puppet using puppet agent -t or mco puppet shell --tail. But it’s important to remember that
this might fail since virtual machine can be used by multiple developers, and only single SSL certificate can exist using existing hostname. If you get an error, rename your
instance and reboot before trying again.

Advanced workflows

1. Custom facts
   When you are developing puppet facts, sometimes it’s very useful to be able to run facter, without any puppet catalog compilation.
   In order to run facter, we need to configure it to point to folder where is our custom code.

For instance, if we were to develop a new custom fact in module base_windows, it’s path should be in base_windows/lib/facter/myfact.rb

So by running facter --custom-dir=C:\Vagrant\puppet-control\vetorzero\base_windows\lib\facter facter would display us all default facts including our own.

2. Hiera configuration testing

It’s possible to invoke hiera, in 2 ways, either using hiera binary, or calling puppet lookup command.
Doing this, we can quickly test what key=value pairs would be used in the hierarchy, without running puppet code.