Automate Multi-Source Code Manager during Puppet Enterprise Install

I’ve been a fan of Puppet Enterprise’s Code Manager since it shipped in PE 2015.3.  When we originally converted our deployments from zack/r10k to CM, we had just a single control repo with all our code environments as branches.  Later, we realized we would streamline our workflow by separating hieradata from control code into its own repo, with different security and its own branches, each consumed by different PE deployments.

In preparing to migrate our PE deployments to Azure, I’ve been developing automation and scripting around the operations involved to avoid misconfigurations as much as possible.  One of the areas I’ve been wanting to automate is Code Manager setup.  Puppet’s documentation on the subject is great, but the current (2016.4.2) PE installer doesn’t allow for CM configuration that involve multiple source repositories or proxy servers.  Complex configurations have to be set up after install.  I figured out how to do this in a streamlined fashion by creating a temporary, local hiera data directory using a relative path at the bottom of the hierarchy and injecting Code Manager configuration data into its common.yaml.

Note that because Code Manager is only configured to deploy per-branch environments to the PE Master’s $codedir, the example I have here deploys our hieradata repository’s branches to $codedir and they will appear as environments named “hiera_<branchname>”.  They can be safely ignored in the PE Console, and should not be assigned as an environment to any node groups.


After Puppet Enterprise setup, place an SSH private key with read-only access to your source repos in the location recommended in the PE documentation.

On the PE master, create /var/tmp/codemgr_key.pp (insert your own private key’s contents):

# /var/tmp/codemgr_key.pp
$codemgr_private_key = '-----BEGIN RSA PRIVATE KEY-----
<private key contents>

file { '/etc/puppetlabs/puppetserver/ssh':
  ensure => 'directory',
  group  => 'root',
  owner  => 'root',
  mode   => '0755',

file { '/etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa':
  ensure  => 'file',
  group   => 'pe-puppet',
  owner   => 'pe-puppet',
  mode    => '0400',
  content => $codemgr_private_key,

Create the Code Manager key by running (on the PE master):

sudo puppet apply /var/tmp/codemgr_key.pp

Make sure to delete /var/tmp/codemgr_key.pp when done if you don’t want to leave your key contents in that directory.

Run this as root on the PE master to set up a Code Manager deployment user and retrieve an authentication token for it:

# Edit as appropriate for your deployment
# Environment variables
CERT="$(puppet agent --configprint hostcert)"
KEY="$(puppet agent --configprint hostprivkey)"
CACERT="$(puppet agent --configprint localcacert)"
SERVER="$(puppet agent --configprint server)"

# Use Puppet's curl
alias curl='/opt/puppetlabs/puppet/bin/curl'

# Install jq
puppet resource package jq ensure=installed

# Make a root .puppetlabs directory for token
mkdir /root/.puppetlabs

# Create deployment user
curl -k -X POST https://localhost:4433/rbac-api/v1/users \
  --cert $CERT --key $KEY --cacert $CACERT \
  -H "Content-Type: application/json" \
  -d '{"login":"deployment", "email":"", "display_name":"Code Manager Service Account", "role_ids": [4], "password":"puppetlabs"}'

# Request an authentication token and store in /root/.puppetlabs/token
curl -k -X POST https://localhost:4433/rbac-api/v1/auth/token \
  --cert $CERT --key $KEY --cacert $CACERT \
  -H "Content-Type: application/json" \
  -d '{"login":"deployment", "password":"puppetlabs", "lifetime":"10y", "label":"PE Master token"}' | \
  jq -r '.token' > /root/.puppetlabs/token

Inject Local Hieradata

# Remove proxy setting if not required.
mkdir /etc/puppetlabs/code/hieradata
cat > /etc/puppetlabs/code/hieradata/common.yaml << COMMONYAML
 private-key: '/etc/puppetlabs/puppetserver/ssh/id-control_repo.rsa'
puppet_enterprise::master::code_manager::proxy: '$PROXY'
 remote: "<clone URL for control repo>"
 prefix: false
 remote: "<clone URL for hieradata repo>"
 prefix: true
puppet_enterprise::profile::master::code_manager_auto_configure: true
puppet_enterprise::profile::master::file_sync_enabled: true
chown pe-puppet:pe-puppet /etc/puppetlabs/code/hieradata/common.yaml
mv /etc/puppetlabs/puppet/hiera.yaml /etc/puppetlabs/puppet/hiera.yaml.old
cat > /etc/puppetlabs/puppet/hiera.yaml << HIERAYAML
 - yaml
 - "nodes/%{::trusted.certname}"
 - common
  - "../../../hieradata/common"

# datadir is empty here, so hiera uses its defaults:
# - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
# - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
# When specifying a datadir, make sure the directory exists.
# Restart pe-puppetserver to pick up new hiera configuration
puppet resource service pe-puppetserver ensure=stopped
puppet resource service pe-puppetserver ensure=running

Configure Code Manager

Perform a Puppet Agent run as root to configure Code Manager:

sudo puppet agent -t

You should see output indicating the setup of File Sync and Code Manager is being done:

Notice: /Stage[main]/Puppet_enterprise::Master::Puppetserver/Pe_hocon_setting[jruby-puppet.environment-class-cache-enabled]/ensure: created
Info: /Stage[main]/Puppet_enterprise::Master::Puppetserver/Pe_hocon_setting[jruby-puppet.environment-class-cache-enabled]: Scheduling refresh of Service[pe-puppetserver]
Notice: /Stage[main]/Puppet_enterprise::Master/Pe_ini_setting[puppetconf environment_timeout setting]/value: value changed '0' to 'unlimited'
Info: /Stage[main]/Puppet_enterprise::Master/Pe_ini_setting[puppetconf environment_timeout setting]: Scheduling refresh of Service[pe-puppetserver]
Notice: /Stage[main]/Pe_r10k::Config/File[r10k.yaml]/ensure: defined content as '{md5}1a51daddd57c58615646202f69847914'
Notice: /Stage[main]/Puppet_enterprise::Master::Code_manager/File[/opt/puppetlabs/server/data/code-manager/]/mode: mode changed '0700' to '0750'
Notice: /Stage[main]/Puppet_enterprise::Master::Code_manager/File[/etc/puppetlabs/puppetserver/conf.d/code-manager.conf]/ensure: created
Notice: /Stage[main]/Puppet_enterprise::Master::Code_manager/Pe_hocon_setting[webserver.code-manager.client-auth]/ensure: created
Info: Computing checksum on file /etc/puppetlabs/puppetserver/bootstrap.cfg
Info: /Stage[main]/Puppet_enterprise::Profile::Master/Puppet_enterprise::Trapperkeeper::Bootstrap_cfg[certificate-authority-service]/Pe_concat[/etc/puppetlabs/puppetserver/bootstrap.cfg]/File[/etc/puppetlabs/puppetserver/bootstrap.cfg]: Filebucketed /etc/puppetlabs/puppetserver/bootstrap.cfg to puppet with sum 49aa06818b9c496279e2543e57bfe6ab
Notice: /Stage[main]/Puppet_enterprise::Profile::Master/Puppet_enterprise::Trapperkeeper::Bootstrap_cfg[certificate-authority-service]/Pe_concat[/etc/puppetlabs/puppetserver/bootstrap.cfg]/File[/etc/puppetlabs/puppetserver/bootstrap.cfg]/content: content changed '{md5}49aa06818b9c496279e2543e57bfe6ab' to '{md5}7144f0a9d7e805620f6a90c8e1a9d55f'
Info: Pe_concat[/etc/puppetlabs/puppetserver/bootstrap.cfg]: Scheduling refresh of Service[pe-puppetserver]
Info: Puppet_enterprise::Trapperkeeper::Bootstrap_cfg[certificate-authority-service]: Scheduling refresh of Service[pe-puppetserver]
Notice: /Stage[main]/Puppet_enterprise::Master::Puppetserver/Service[pe-puppetserver]: Triggered 'refresh' from 44 events
Notice: Applied catalog in 43.61 seconds

At this point, you should be able to perform your first deployment with Code Manager:

puppet code deploy --all --wait

Follow the Code Manager documentation for further instructions on settings up webhooks, etc.

Thomas Woodrow Harden: 1948-2016

Another great person has left us in 2016. My father, Thomas Woodrow Harden, unexpectedly passed away on his 46th Father’s Day, June 19th, after contracting an infection during a bicycling tour that his body couldn’t successfully overcome. Tom had heart-related problems that he’d been managing with medical care for several years.

My Dad was one of my best friends, someone who I was never afraid to talk to about almost any subject. If you knew him, you know he held an opinion about many things, and his were right. 🙂

Tom died doing two things he loved: bicycling and visiting with family. When he told me he planned to “Ride the Fault Line” and visit his relatives living in Sikeston, Missouri, he immediately invited me to ride it with him. In early June the previous two years, we had ridden a 3-day ride in New York State together, but a road trip to Sikeston for a week-long ride was more than my responsibilities would allow. I’ll regret not going on that trip for a long time.

My Dad loved long bike tours, with or without support. In recent years he went on supported and self-contained tours organized by Adventure Cycling, and made many new friends. This January he purchased a new sport bike for rides where he didn’t have to carry all his gear. He enjoyed building and customizing bikes, and through our time together he impressed his love of cycling on me. I’ll never forget when I was 13 and he bought me my first 12-speed road bike. Another highlight was when we rode the 5-day Bike Virginia tour in 2005 together. It was my first taste of a multi-day supported bike tour, and it took me some work to prepare for it. It was grueling for me, but at the end of those days I enjoyed relaxing and camping with my Dad. (I love camping because him, too.) We took an optional day in Lexington to rest our “parts”, exploring much of the town on foot, visiting the colleges there, and watching “Batman Begins” at the local theater. I’d be remiss if I didn’t mention that I can’t recall a time when my Dad didn’t want to go see a movie I was interested in. At the end of that ride, a 72-mile trip back to Roanoke, I knew I would be doing this again. And I wanted to do it with my Dad.

I mentioned my Dad loved visiting with family. Thankfully, he and my Mom, Rhonda, along with my sister, Candace, and her kids had come to visit my family in Harrisburg for Memorial Day weekend two weeks prior to his last ride. We shared some great meals and outdoor activities together, and he got to spend time with his four grandchildren. That was one of the first things Candace told me as she broke the news of his passing over the phone to me and tried to console me.

My Dad grew up Baptist, but became an agnostic. I believe this was for two fundamental reasons. First, he witnessed the horrors of war as Army soldier in Vietnam in his early 20s; second, his religious upbringing excluded the theory of evolution, which he firmly believed. Even so,, he never discouraged our faith, allowing Candace and I to be raised Catholic. He would occasionally attend holiday masses with us, but many times would take issue with the priest’s homily. I regretted that my Dad didn’t share our faith, but we found common ground with respect to evolution and other scientific matters. Even though my Dad wasn’t religious, I believe his is the kind of soul that will live forever in heaven.

Another thing important to my Dad was cooking. He impressed this upon me in several ways: I cook Chicken Kiev each Christmas Eve after being inspired by his attempts (I still haven’t achieved 0% butter leakage); I love to grill with charcoal; and I always prefer to cook dinner for my family. In recent years, Tom sang the praises of Cook’s Illustrated (the magazine behind “America’s Test Kitchen”) because they were scientists in the kitchen like I believe he was. Recently he held a chili cook-off at his house where family and friends brought their creations and we got to enjoy all of them. I don’t recall him holding a vote at the end of dinner, because I think he wanted everyone to feel like they won. My Dad cooked with rich ingredients unabashedly, because he had every intent to work off those calories on the bike.

Tom also loved dogs. We never had a dog growing up, but he was always keen to greet and play with any that were around. In our rides together, he sometimes made pit stops to visit with the “free-ranging mongrels” he hoped to encounter on the road. In his last few months he had started to volunteer at a local dog shelter, and told me a few stories to tell about the dogs he was working with. I could tell it was a post-retirement hobby he was happy to take on.

In the early 80’s, my Dad took up a new pastime when he acquired a 22-foot MacGregor sailboat. We sailed as a family over the years in the Memphis, TN area and later in upstate New York. Our most frequent outings were on Canandaigua Lake, where our captain honed his first crew (us) and his knowledge of sailing grew. He customized his boat, and would sometimes trailer and sail it solo. Later he would join the crew of the 35-foot “Kemah” sailing out of the Rochester Yacht Club on Lake Ontario with our neighbors, Paul and Rita. This is another of Tom’s loves that grew on me: I tried my hand at sailing my own boat a few years ago, but after a number of outings (some with my Dad) I found that I make a much better first mate. He was still a fine captain.

Of all the things he loved, my Dad loved my Mom most of all. I know they had their arguments and rough patches, and he needed (and took) a lot of space by working jobs that required travel as well taking his own solo recreational trips (usually on bicycle). However, they were happily married for nearly 47 years, enjoying their last 20+ years as empty-nesters and the last few as retirees. They loved to travel and formed a lot of memories outside their homes. My Mom is going to need more support now that her Tom won’t be returning home, and I plan to help her as much as she needs.

I’ll miss conversations with my Dad, whether on the phone or on the bike, about work (we both worked in IT-related fields), life, parenting, cooking, cycling, or whatever he wanted to talk about. My Dad could be curmudgeonly and short-tempered (that rubbed off on me, too), but if you gave him time and listened to what he had to say, he’d listen to you, too. He relished good arguments, and occasionally changed his point of view. (Some would say “rarely”.)

To honor my Dad, I’m planning to ride one of his favorite tours, the 2017 Bon Ton Roulet, in his memory (and in his jersey) to raise money for the American Heart Association. I’m confident he’ll be riding every mile with me in spirit.

Mini-Review: “Star Wars: The Force Awakens” (Spoilers)

The Shirt AwakensI went and saw Star Wars: The Force Awakens today with my two sons. Coincidentally, I was 8 in 1978 when I saw the original Star Wars in a theater; today, my 8-year-old saw his first (new) SW movie in a theater. It was also his 15-year-old brother’s first. Having avoided spoilers (which I thought would be impossible these days), I loved almost every minute of it, knowing after the opening act that we were seeing a highly nostalgic and derivative story. It was worth the ride. Many faces were fresh, the acting was enjoyable, and the effects looked great. I’m hopeful that we experienced the long-overdue bridge to new stories in the SW universe, and that we’ll get a steady trickle of new tent-pole movies and short-form, long-arc series.

I love the new characters Rey, Finn, Poe, and BB-8. I was disappointed that R2-D2 had a small cameo. I was grateful that Han was a main character, although I felt cheated that we got very few details from his last 30 years. Perhaps there are more details in the novelization. There were enough plot holes to frustrate me after taking it all in; I’ll consider them meat for the writers to chew on. The climactic scene between Han and Ren could have been much better; Han’s fate wasn’t surprising.  Chewie’s action scenes were. BB-8 provided great comic relief.

I really enjoyed watching Rey and Finn, although Rey seemed entirely too familiar with The Force and Finn too familiar with a lightsaber. Ren alternated between god-like powerful and inept; perhaps that was JJ Abrams’ way of exposing his supposed inner conflict. Poe was just cool.

Will I go see this in the theater again? Probably not. But I will buy the Blu-Ray on the first day of release. Consider me tuned back in to the Star Wars saga.

Is Blogging Back?

Like many others, I’ve neglected to blog in the last few years due to the rise of centralized social media sites. I don’t do Facebook, but I’ve been active on Google+ and Twitter.

Thinking about resolutions for 2016 and testing out the WordPress iOS app, I think it’s time to re-skin this site as more than just an online business card and start writing again. In the last year I’ve transitioned into tech at work that is putting me back into the mode of consistently having more than enough opportunities to learn new stuff, and I’ll share thoughts on that. I’m also starting to do some open-source development in the Puppet community.  Some of those thoughts may be here, too. Let’s see.

For Sale: ADA MB-1/B500B Bass Bi-Amplifier Solution

I’ve never used my ADA MB-1/B500B combo to its potential, and with the plethora of smaller, lightweight bass heads, I’ve recently moved to another bass amp.  I’m pleased to offer my ADA solution for sale.  It’s in great condition, has been well cared-for, and sounds great.  The solution includes:

  • ADA MB-1 Bass PreAmp (v2.05 firmware with original documentation)
  • ADA B500B Bass Bi-Amplifier (with original documentation)
  • Tripp-Lite ISOBAR 12 Ultra surge suppressor
  • CDM Manufacturing 6U Rack Case
  • Patch cables

I’ll sell the solution as-is for $600.  If you want just the two ADA pieces, I can put them in a less-featured 4U rack case and sell them for $450.  If you also want the surge suppressor mounted in the 4U case, that would cost another $50.

Contact me through this posting, or “aharden” at Gmail.






Invisalign – 42 Trays to Go

It’s been a while since I’ve blogged here, or at all. Breaking the silence. This is going to become a journal again…

I started an Invisalign regimen with Hilton-Diminick Orthodontic Associates this week. Dr. Hilton is Ryan’s orthodontist as well. My bite shifted significantly last year, coincidentally when I started my weight loss journey. After waiting to get additional orthodontic coverage on my dental insurance, I signed up for the process and went through the high-res scan of my teeth. It was amazing to see them rendered in 3D and I was surprised at the preliminary model of straightening and bite correction the scanning workstation rendered independently.

After a few weeks, Dr. Hilton reviewed the treatment course with me and interactively showed me the final 3D models for the 42 steps recommended. I committed to the process and made a significant down payment to get the initial set of trays fabricated. At my appointment this week, I got my cases, my first two sets of trays and some instruction.

The trays go in and come out without too much trouble. My mouth and teeth are now more consistently clean thanks to the extra brushing and rinsing required to safely use the Invisalign trays. After two full days of wearing them the recommended 22 hours (or more), the initial soreness is gone. This is a very manageable process and hasn’t been too terribly inconvenient yet. It will definitely help my dieting; other than water, you’re not supposed to eat or drink anything while you’re wearing the trays. That means I’ll be drinking much less coffee and tea, and grazing during the day is done. I’m making up for some of that by not skipping breakfast, and making each meal (and the tooth brushing afterwards) count. This won’t be all that bad.

Cue Sheet for Lower Paxton/West Hanover/Hummelstown Circuit Bike Ride

This is a pleasant ride through the suburbs of Dauphin County. Enjoy!

Cue Sheet for Lower Paxton/West
Hanover/Hummelstown Circuit

Start: Diocese of Hbg Bldg
EB Union Deposit Rd
@Newside Rd interchange
(east of I-83)

R Newside Rd
R Page Rd
L Lyters Ln
R Conway Rd
L Nyes Rd
R Willoughby Rd
R Union Deposit Rd
L Copperstone Rd
R Red Top Rd
L Hunters Run Rd (no sign)
R Deaven Rd
R Yorkshire Dr
R Devonshire Hts Rd
R Oak Grove Rd
Becomes N Hoernerstown Rd
L Red Top Rd
R Stoudt Rd
R Grandview Rd
Becomes N Hanover St
L E 2nd St (after underpass)
R Cameron Ave
R E Main St
Becomes Bridge Rd
R Pleasant View Rd (at In Gear)
L Old Derry St (red arrow)
R S Nyes Rd
L Derry St
R Lawnford Way
L Lancaster St
R N 50th St
R Harvest Dr
L Spring creek Rd
R Twin Lakes Dr
R Union Deposit Rd

Total: 22.4 mi

Google's Goal: Kill Open Syndication

The decommission of Google Reader isn't just about getting people to stop using RSS.  It's the first battle in a war against making it feasible for web sites to syndicate agnostically using RSS technologies.  Google wants content providers to pick sides in their war against Facebook, Twitter, etc.  FeedBurner will be shut down next.  Sites will be forced to syndicate directly into social networks as their RSS feeds become less used and RSS publishing/metric tools go away.  Web usability, users, and content providers lose; the social networks win.  #wp

Google+: View post on Google+

Post imported by Google+Blog. Created By Daniel Treadwell.

Weighed in at 207 today

I've been counting calories for a month using, and haven't had any binges. My daily calorie allowance is now 1910. I think I will be able to get to my goal weight of 200 lbs in another month, and may get under 200 by my goal time of June 1st. #wp

Google+: View post on Google+

Post imported by Google+Blog. Created By Daniel Treadwell.

I've been using RSS readers for over a decade to follow blogs, music news, and…

I've been using RSS readers for over a decade to follow blogs, music news, and tech industry news.  Google Reader has been the most convenient platform for this for many years.  I was a Bloglines refugee who wanted to use a reader that supported the Atom format.  Forget GReader's interface and the debacle when it was "Plusified" — many third-party apps use the Google Reader API, which allowed me to do my feed reading with non-Google apps.  I agree with Mike's point that they should have mined it for advertising (we users would welcome it vs. the platform going away).  Of all the moves that have been made since Larry Page took over, this is the most blatantly out of touch with the user community I'm a part of.  Guess it's time to install Planet or Venus and change my feed-reading habits.  Thanks for nothing, Google.  #wp

Reshared post from +Mike Elgan

Five thoughts on Google Reader's death sentence.

Google announced that it would kill Google Reader starting July 1 as part of its "spring cleaning."

Although I broadly applaud the Larryfication of Google, including the consolidation and focus that requires sometimes unpopular shutdowns, this one surprises me, and I have five points to make about it:

1. I would guess that although RSS in general and Reader in particular have fallen out of favor with the general public (in favor of Twitter, etc.), is Google aware than nearly all tech journalists and bloggers who cover Google rely heavily on RSS and most of them on Reader? Google is taking away the main tool the tech press uses to keep up with news, and its unclear what impact this will have on Google's relationship with the press.

2. Presumably Google would prefer that Reader users use Google+ and the features in Google News that enable you to tailor the news you see for content discovery, but it may have the opposite effect, making them believe that Google can't be relied upon to keep services going.

3. And this is a bit of an announcement, but I am preparing to do an experiment whereby I use only Google products for one month, hardware and software and service. My experiment will take place before the Reader shutdown, but I am planning to rely on Google Reader heavily during that month. I'm not sure the experiment would be possible afterwards.

4. It seems to me that Google never tried to monetize Reader. With advertising, it seems like it could have been self-sustaining, because….

5. Reader seems like an ideal harvester of "signals" for advertising and other efforts by Google. It tells Google exactly what users are interested in.

Anyway, I'm surprised by Google's decision to kill Google Reader; it seems to me that the benefit of keeping journalists, power users, developers and other Reader users happy would outweigh the costs of keeping Reader alive.

What do you think?

Google+: View post on Google+

Post imported by Google+Blog. Created By Daniel Treadwell.