Troubleshooting Puppet 8 with PuppetDB: The val Bug and Other Gotchas

Posted on by

This year I moved all my lab nodes to Rocky 9 and Puppet 8, aside from moving a lot of services to Kubernetes. As I move along, I blog about it, like for instance about the new monitoring with PLG. In the past I also used Nagios, because that has it’s values too. But instead of managing Nagios manually, which is immensely tedious, I used Puppet via PuppetDB to populate it automatically.

Puppet happens to be my specialty since many years and I am writing custom Puppet modules since 2010. Amongst them my puppet_cd module, which I am rewriting at the moment to adopt to the lastest use cases. Including integrating PuppetDB.

This weekend I decided to return to including Nagios in addition to PLG for some reasons, and naturally needed PuppetDB. This application is not necessarily required for running a Puppet ecosystem, but it is extremely helpful for storing configurations, cross-node-data-sharing, managing exported resources and so on. Pretty nifty thing. And in the past it was a flash to set up and going. Except this time, with Puppet 8. after enabling PuppetDB (see below) , it threw an error 500 “undefined local variable or method val' for Puppet::Util::Puppetdb:Module” and no puppet run went through. Almost nothing on the web, only vague things about error 500 ( which per se is quite common in various scenarios). Solving this took me 2 days, and here is some clues for you so save that time. Let’s first look at how PuppetDB normally is put in place.

Installing and enabling PuppetDB

There are several ways to do this, one is to use packages . Well, I am using my own module for it, but that follows strictly those steps and will be soon merged with the main puppet_cd module for these topics belong together. But in short here are the manual steps:

  • Install Puppet and your Puppetserver ( you mot likely will have that done already of you read this)
  • use Puppet to install PuppetDB ( do not yet start the service):
sudo puppet resource package puppetdb ensure=latest
  • configure a database. In a production environment, PostgreSQL is highly recommended and in fact the only external database supported. that can be on the PuppetDB host or externally, I prefer the latter because likely there is one already and there is no need to manage a fleet of it. Mine is managed via Puppet too, including managing roles and databases, but you might have to go the manual steps in accordance to your setup. You need an empty database, a user and password for it and a proper entry in your pg_hba for this to work. Once this is in place, configure your PuppetDB host. A full list of settings is here , but in general all the files you need are already there in /etc/puppetlabs/puppetdb/conf.d:
    • auth.con
    • config.ini
    • database.ini
    • jetty.ini
    • repl.ini

For now we need only to edit database.ini:

[database]

# The database address, i.e. //HOST:PORT/DATABASE_NAME
subname = //localhost:5432/puppetdb

# Connect as a specific user
username = <username>

# Use a specific password
password = <password>

# How often (in minutes) to compact the database
gc-interval = 60

You might likely want to set TLS to have that connection encrypted, but for now this helps to be going.

  • Enable and start the PuppetDB service now
sudo puppet resource service puppetdb ensure=running enable=true

Your service should be up and running, verify it: 

sudo service puppetdb status
Redirecting to /bin/systemctl status puppetdb.service
● puppetdb.service - puppetdb Service
     Loaded: loaded (/usr/lib/systemd/system/puppetdb.service; enabled; preset: disabled)
     Active: active (running) since Tue 2025-10-28 13:51:38 CET; 1h 34min ago
   Main PID: 29812 (java)
      Tasks: 68 (limit: 4915)
     Memory: 410.5M
        CPU: 1min 21.331s
     CGroup: /system.slice/puppetdb.service
             └─29812 /usr/bin/java -Xmx192m -Djdk.tls.ephemeralDHKeySize=2048 -Dlogappender=F1 -XX:+CrashOnOutOfMemoryError>

Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,436 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNA>
Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,436 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNA>
Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,437 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNA>
Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,437 |-INFO in ch.qos.logback.core.model.processor.ImplicitMod>
Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,444 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender>
Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,444 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender>
Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,444 |-INFO in ch.qos.logback.core.model.processor.AppenderRef>
Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,444 |-INFO in ch.qos.logback.core.model.processor.DefaultProc>
Oct 28 13:51:36 fm002.confdroid.com puppetdb[29812]: 12:51:36,444 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@5>
Oct 28 13:51:38 fm002.confdroid.com systemd[1]: Started puppetdb Service.

As far as PuppetDB is concerned, now you are good and ready. But there is far more to it and here is where the bug hit me. First of all though: Can and should PuppetDB be running on your Puppetmaster or on a separate node?

Well, it can be on your Puppetmaster, yes. But assuming you have a real production environment to manage, you will likely have more than one Puppetserver for redundancy. in that scenario, having your PuppetDB separate is a very good option.

So you may well choose to use a separate node right away to avoid a big mess in future when expanding. No matter how many Puppet servers you have, each and any of them must be configured to use PuppetDB. There are 4 critical components:

  • A plugin called Puppetdb-termini
  • a config file for the details about the PuppetDB server
  • a config file for the routing
  • A change in the puppet.conf file

The plugin is installed with a package:

sudo dnf install puppetdb-termini

When using Puppet, just add it to the list of required packages for the puppetserver.

The configuration for the PuppetDB server: 

vi /etc/puppetlabs/puppet/puppetdb.conf
[main]
server_urls         = https://puppetdb.example.net:8081
soft_write_failure  = false

This two liner is all what’s needed, but here lies a trap. More about it below.

Finally the routing file:

vi /etc/puppetlabs/puppet/routes.yaml
---
master:
  facts:
    terminus: puppetdb
    cache: yaml

That is all but without the file it won’t work.

Once these things are in place. restart you puppetserver: 

sudo service puppetserver restart

Finally the change in the puppet.conf

vi /etc/puppetlabs/puppet/puppet.conf

in the [server] section, find 
<storeconfigs = false>
and replace it with 
storeconfigs = true
storeconfigs_backend = puppetdb

This tells your puppet server to use PuppetDB for storing configurations. Restart that service once more and now you should be up and running. But here comes the nasty bug, introduced in Puppet 8:

At the next puppet run, you might see 

undefined local variable or method `val' for Puppet::Util::Puppetdb:Module

And none of the coming puppet runs will pass.

This is because the Puppet ruby code does some verifying and if the configuration is not right, it should tell where the problem is. But due to a bug in /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb around line 50, containing this: 

 def self.to_bool(value)
    case value
    when true, "true"; return true
    when false, "false"; return false
    else
      raise ArgumentError.new("invalid value for Boolean: \"#{val}\"")
    end
  end

That should say “value”, not “val”. The bug only comes through, when your puppetdb.conf looks like this: 

[main]
server_urls         = https://puppetdb.example.net:8081
soft_write_failure  =

Note the missing boolean, there should be true or false, but there is none. In my case that happened because a parameter I had given as place holder was misspelled and as such the file did not contain that value. Because of the bug, instead of telling the problem, it comes up with that obscure 

undefined local variable or method `val' for Puppet::Util::Puppetdb:Module

If yoo go and change that section above to include 

 def self.to_bool(value)
    case value
    when true, "true"; return true
    when false, "false"; return false
    else
      raise ArgumentError.new("invalid value for Boolean: \"#{value}\"")
    end
  end

and restart the puppetserver service, suddenly you get a proper error:

Server Error: invalid value for Boolean: " "

and that finally got me to the missing value. I fixed my puppet module to have the spelling error corrected, and suddenly all went well as the boolean value was then set as needed.

When I went and reverted that ruby code above, the problem still was gone, but I never had figured out where this is coming from. So I will see if I can bring that to the attention of the puppet folks to get it fixed.

So naturally my Puppet modules do the required tricks now and I am gonna merge them into one ( puppet_cd) and will update the Readme to explain this all nicely.

TL;DR: If your puppetdb.conf on the puppetmaster is missing the boolean value for
soft_write_failure = , with puppetdb activated you will get this error. Add either True or False and you are set.

About The Author

sunrider

See author's posts

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

5 × 3 =