Control Windows with Salt

This long article explores Windows management with Salt.
First, it shows you how to automatically install software to Windows, control Windows boxes behind NAT and firewall and remotely run arbitrary PowerShell commands on Windows.

$ sudo salt '*' pkg.install gedit,firefox,steam,vlc
$ sudo salt winslave cmd.run 'Get-ChildItem C:' shell='powershell'

Later, more complicated features are looked at.

This article has been updated with Ubuntu 18.04 LTS and latest Windows 10 information.
Windows is a popular platform for games, and sometimes you need it for some specific software. After all, it’s probably the most popular desktop.  Even though Windows command line has been improving with PowerShell, it still lacks package manager. Salt makes it easier to administer Windows boxes.
Prerequisites: Following this article requires familiarity with Linux command line, Salt master-slave pull architecture and some knowledge of Windows.
This article is written from memory. It’s long and not troughoutly tested. Instead, it points some interesting directions for managing Windows. Two Windows slaves are used in this article, ug and winslave.
If you need a sample Windows installation for testing, http://modern.ie has no-cost VirtualBox images of Windows 10. If you’re getting tired of Windows, Xubuntu Linux might help.

Fresh Version on Salt Master

Start with a working Salt master-slave setup on Linux.
Update: Ubuntu 18.04 LTS has Salt version 2017.7.4 Nitrogen, which works with Windows slaves. If you don’t have a specific need for a newer version, you can use the one from default repositories instead of installing latest version.
All versions of Salt are listed on repo.saltstack.com.

Latest versions of salt work better on Windows. Master salt version must always be same or newer than slaves. So, on salt master

$ wget https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest/SALTSTACK-GPG-KEY.pub
$ sudo apt-key add SALTSTACK-GPG-KEY.pub  # New trust

Adding a new key as trusted means a new trust relationship.

$ echo "deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/2018.3 xenial main"|sudo tee /etc/apt/sources.list.d/saltstack.list

$ sudo apt-get update
$ sudo apt-get -y install salt-master salt-minion
$ sudo systemctl restart salt-minion
$ sudo systemctl restart salt-master
$ sudo salt '*' test.ping

Check your master ip address or public, fully qualified domain name. In the next step, slave needs to know where to connect.

$ hostname -I

Make Windows a Salt Slave

On Windows (to-be) slave, download and install the same Salt version salt-minion package. You probably want Python 3 and amd64 version.
Update: All versions of Salt Minion for Windows on Salt homepage
e.g. https://repo.saltstack.com/windows/Salt-Minion-2018.3.0-Py3-AMD64-Setup.exe
Add your master IP address and minion id when asked. Minion id must be different from other minion ids.

Accept Windows as a Slave

List all keys, accepted and unaccepted.

$ sudo salt-key

Accept slave key

$ sudo salt-key -A

Test

$ sudo salt '*' test.ping

At least in my testing with virtual machines, Windows slaves are much slower than Linux slaves. If you get a timeout, just try again.

Enable Salt Windows Software Repositories

This is the one-time configuration so that packages are easy to install.
Update: On newer salt, you must give “salt” group write permission to /srv/salt/win/ on master.

$ sudo mkdir /srv/salt/win
$ sudo chown root.salt /srv/salt/win
$ sudo chmod ug+rwx /srv/salt/win

You must have git installed to update your git repos. If only there was a way to install git on all machines on my network…
$ sudo apt-get -y install git

On Salt master

$ sudo salt-run winrepo.update_git_repos
$ sudo salt -G 'os:windows' pkg.refresh_db

Install Software – Firefox, Steam, Inkscape…

Install a single package

$ sudo salt '*' pkg.install vlc

Or many

$ sudo salt '*' pkg.install gedit,firefox,steam,vlc

Of course, you can also use pkg.installed in your states (idempotent configuration).

Run PowerShell

You can do most Windows things by running PowerShell. It’s not bash, but it’s something.

$ sudo salt ug cmd.run 'Get-ChildItem C:' shell='powershell'
ug:
 Directory: C:\Windows\system32\config\systemprofile
 Mode                LastWriteTime         Length Name
 ----                -------------         ------ ----
 d-----        9/29/2017   6:46 AM                AppData

Well done, you can now control Windows with Salt.

Run Salt Locally on Windows

On Windows desktop, useful tools menu opens with ugly-X.
Choose “Windows PowerShell (Admin)”.
Windows I had was using US keymap. Wonder what’s ‘setxkbmap fi’ in PowerShell?

> Set-WinUserLanguageList -LanguageList fi-FI -Force

It really works. You can even write “äöäö”.
They have made ‘ls’ work on Windows. And we can run ‘salt-call –local’, a masterless, standalone setup suitable for quick testing.

> c:
> cd /salt/
> ls
> ./salt-call --local test.ping

Weirdly enough, sometimes salt-call seems to be in PATH, so I can use it from any folder in PowerShell. Also ‘git’ seems to be available everywhere.

> salt-call --local pkg.install git

To see some debug output

> salt-call --local pkg.install git -l debug

Maybe you could download or git clone your states here and run them standalone. But of course, a real master is better setup.

> git clone https://github.com/terokarvinen/sirotin.git
> cd sirotin
> salt-call --local --file-root srv/salt/ --pillar-root srv/pillar/ state.highstate --state-output terse -l warning

Prints quite a few error messages, as these states are for Linux. But we can see that it probably works, you could write and run standalone states on Windows.

Install Even More Packages on Chocolatey

The 250 pkg.install packages are easier to verify – just look at repo-ng how they are made. But Chocolatey has over 5000 packages. So let’s install chocolatey.

$ sudo salt ug pkg.install chocolatey
$ sudo salt ug chocolatey.install putty.install

After a while, Putty SSH client is available on Windows desktop.
There are a lot more installable packages in Chocolatey Gallery.
While it all this makes Windows administration easier, it’s definitely no match to apt-get. For example, using chocolatey to install one of the most popular of its packages, googlechrome, resulted in an error.
Let’s install some more packages

$ sudo salt ug chocolatey.install sysinternals
$ sudo salt ug chocolatey.install classic-shell
$ sudo salt ug chocolatey.install cygwin
$ sudo salt ug chocolatey.install cyg-get

If you get a timeout, it’s probably just the installer taking over 5 seconds. You can look at jobs running on minions with

$ sudo salt-run jobs.active

If this annoys you, just write an idempotent SLS state. You can have salt do the work while you’re taking a break. And it can do this work on many, many computers.

Windows State for Installing a Lot of Apps

Update: choco idempotent installation requires a funny syntax with salt 2017.7.4 Nitrogen and Windows 10.
Running this will take a while…

$ cat /srv/salt/deskwin/init.sls
## This state is not completely tested
deskwin:
  pkg.installed:
    - pkgs:
      - gedit
      - firefox
      - steam
      - vlc
      - inkscape
      - git
      - winscp
      - putty
      - classicshell
      - python3_x64
      - adobereader
      - chrome
      - libreoffice
      - pandoc
      - pidgin
      - thunderbird
chocolatey:
  pkg.installed
choco:
  chocolatey.installed:
    - pkgs:
      - sysinternals
      - cygwin
      - cyg-get
      - syncthing

When you mention this deskwin in your /srv/salt/top.sls, you can apply highstate

$ sudo salt '*' state.highstate

You might want to check the status every now and then, as this will take a while.

$ sudo salt-run jobs.active

By the way, now that we have cyg-get, we can install cygwin packages with a command and some UAC prompts.
cygwin$ cyg-get.bat vim

Windows Server Roles Not Supported in Many Windowses

Update: This “Server Roles” part describes some failed tries on server roles. Feel free to skip it. Windows licensing requirements limit available images, and the one I had for this test did not support roles.


>>> Skip to “What Next?” >>>
Server roles are only available on Windows servers 2008 R2 and later. They are not available on desktop Windowses.
Windows daemons can be installed using “roles” with salt.modules.win_servermanager. For example, this does not work on the desktop we are using

$ sudo salt ug win_servermanager.list_installed
ug:
    'win_servermanager' __virtual__ returned False: Failed to load win_servermanager module: ServerManager module not available. May need to install Remote Server Administration Tools.
ERROR: Minions returned with non-zero exit code

Windows Server 2016 Vagrant Box

On Ubuntu 16.04 packaged Vagrant 1.8.1, most Windows boxes don’t work.
I have a lot of Windows licenses at work. You must yourself evaluate if you have the license to use the Windows image in question. It goes without saying that on the Linux side, we don’t have to spend time counting per-seat and per-server and per-core and per-year licenses…

$ vagrant init mwrock/Windows2016
$ vagrant up
/usr/share/vagrant/plugins/communicators/winrm/shell.rb:9:in `require': cannot load such file -- winrm (LoadError)

Also, winrm plugins can’t be installed on that version of vagrant.
Destroy the machine if it’s still hanging there, with virtualbox GUI if needed. Remove old vagrantfile.

$ vagrant destroy
$ rm Vagrantfile

To fix, install newer Vagrant. As a package is installed from the web, a new trust relationship is created.

$ wget https://releases.hashicorp.com/vagrant/2.0.3/vagrant_2.0.3_x86_64.deb
$ sudo gdebi -n vagrant_2.0.3_x86_64.deb # Do we trust this file?

Now, we can install Windows Server 2016 box

$ vagrant init mwrock/Windows2016
$ vagrant up

A window pops up. We can login as vagrant, password vagrant.

Install Salt Minion to Windows Server

Just like before, we Google or DuckDuckGo “salt windows install” and install the minion. Use your master ip address and give the slave a unique minion id (“sewi”).
https://repo.saltstack.com/windows/Salt-Minion-2018.3.0-Py3-AMD64-Setup.exe
Accept the slave on your master and test.

$ sudo salt-key -A
$ sudo salt '*' test.ping
sewi:
    True
ug:
    True

If you don’t get an answer immediately, try a couple of times.When your new slave answers, you have a new Windows server to play with.

Trying to Add a Role

Now that we have a Windows Server slave, we can start using roles.

$ sudo salt sewi win_servermanager.list_installed
sewi:
    ----------
    FS-SMB1:
        SMB 1.0/CIFS File Sharing Support
    FileAndStorage-Services:
        File and Storage Services
    NET-Framework-45-Core:
        .NET Framework 4.6
    # ...

About 15 roles enabled.

$ sudo salt sewi win_servermanager.list_available|head
sewi:
    Display Name                                            Name
    ------------                                            ----
    [ ] Active Directory Certificate Services               AD-Certificate
        [ ] Certification Authority                         ADCS-Cert-Authority
        [ ] Certificate Enrollment Policy Web Service       ADCS-Enroll-Web-Pol
        [ ] Certificate Enrollment Web Service              ADCS-Enroll-Web-Svc
        [ ] Certification Authority Web Enrollment          ADCS-Web-Enrollment
        [ ] Network Device Enrollment Service               ADCS-Device-Enrollment

Over 260 lines of roles available.
Windows slaves are sometimes just a bit slow to answer. To make it possible to query available roles instantly, let’s store them to a local text file.

$ sudo salt sewi win_servermanager.list_available>roles-avail
$ grep -i IIS roles-avail
    [ ] Web Server (IIS)                                    Web-Server

As documentation of salt.modules.win_servermanager and salt.states.win_servermanager show, we can also install roles. Obviously, I recommend Apache as a web server, but let’s try IIS to test roles.
The built in Internet Explorer has some bug that makes it difficult to test anything on localhost. Curl did not want to install with pkg.install. But we can use salt’s built in http.query.

$ sudo salt sewi http.query 'http://terokarvinen.com/'
sewi:
    ----------
    body:
        <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        # ...

So when it works, it dumps the HTML to screen.
Windows Server doesn’t run the web daemon IIS yet, so the port is closed and we get “connection refused”.

$ sudo salt sewi http.query 'http://localhost/'
    # ...
    ConnectionRefusedError: [Errno 10061] Unknown error

Let’s change that by installing IIS role.

$ sudo salt sewi win_servermanager.install Web-Server
sewi:
    Passed invalid arguments to win_servermanager.install: string indices must be integers

That would have been too easy… Maybe with all the subroles

$ sudo salt sewi win_servermanager.install Web-Server recurse=True

Don’t worry about timeout “Minion did not return. [Not connected]”. Just use ‘sudo salt-run jobs.active’ to see if it’s still running. Once jobs.active output is empty, the role installation is done.
The machine decided to reboot itself. After the boot, a popup: “We are adding a new feature to Windows”. Maybe next time, for clarity
However, it still was not there, not even listed with win_servermanager.list_installed.
To debug: Start Menu, PowerShell – right click, Run as Administrator.

> cd /salt
> ./salt-call.bat --local -l debug win_servermanager.install Web-Server recurse=True

Now, at least it claimed to install with a long ASCII bar. However, an all too familiar error:

[DEBUG   ] LazyLoaded win_servermanager.install
[DEBUG   ] PowerShell: Import-Module ServerManager; Install-WindowsFeature -Name Web-Server -IncludeManagementTools -Inc
ludeAllSubFeature  -WarningAction SilentlyContinue | ConvertTo-Json
[DEBUG   ] LazyLoaded cmd.shell
[INFO    ] Executing command 'Powershell -NonInteractive -NoProfile "Import-Module ServerManager; Install-WindowsFeature
 -Name Web-Server -IncludeManagementTools -IncludeAllSubFeature  -WarningAction SilentlyContinue | ConvertTo-Json"' in d
irectory 'C:\Users\vagrant'
[ERROR   ] Command 'Import-Module ServerManager; Install-WindowsFeature -Name Web-Server -IncludeManagementTools -Includ
eAllSubFeature  -WarningAction SilentlyContinue | ConvertTo-Json' failed with return code: 1
[ERROR   ] output: Install-WindowsFeature : The request to add or remove features on the specified server failed.
Installation of one or more roles, role services, or features failed.
The source files could not be found.
Use the "Source" option to specify the location of the files that are required to restore the feature. For more
information on specifying a source location, see http://go.microsoft.com/fwlink/?LinkId=243077. Error: 0x800f081f
At line:1 char:30
+ ... verManager; Install-WindowsFeature -Name Web-Server -IncludeManagemen ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (@{Vhd=; Credent...Name=localhost}:PSObject) [Install-WindowsFeature],
    Exception
    + FullyQualifiedErrorId : DISMAPI_Error__Failed_To_Enable_Updates,Microsoft.Windows.ServerManager.Commands.AddWind
   owsFeatureCommand
{
    "Success":  false,
    "RestartNeeded":  1,
    "FeatureResult":  [
                      ],
    "ExitCode":  1000
}
[DEBUG   ] Json not returned
# ...
Traceback (most recent call last):
  File "C:\salt\bin\lib\site-packages\salt\cli\caller.py", line 212, in call
    ret['return'] = func(*args, **kwargs)
  File "c:\salt\bin\lib\site-packages\salt\modules\win_servermanager.py", line 215, in install
    if out['FeatureResult']:
TypeError: string indices must be integers

Maybe we’re lacking Remote Server Administration tools.

$ grep -i remote roles-avail
    # ...
    [ ] Remote Server Administration Tools                  RSAT
$ sudo salt sewi win_servermanager.install RSAT recurse=True

Finally tried with a plain two line PowerShell locally. Surprise, the box image doesn’t support role installation at all.  Next try with another Windows Server installation.
From this role text, it can be concluded that Vagrant Cloud has some non-working Windows images that don’t support software installation with roles.

What Next?

How about writing some Salt states for Windows?
Choose something to pkg.install from 250 applications in repo-ng? Or some more to install from 5000 applications in choco gallery?
Or just getting back to Xubuntu…

Updated: this article has been updated multiple times.