Connecting a Remote Cluster with Stockfish Chess Engine

Part II: How to setup a connection to compute cluster engine

In the?second part 2 I?will show how to connect to ChessBase and other UCI software. ChessBase is a personal, stand-alone chess database that has become the international standard for top players, from the World Champion to the amateur next door.?

If you haven't read the part I "Building a Remote Cluster with Stockfish Chess Engine" I recommend going here first.

As said in the first part, Chessbase only accept exe files; therefore a “middleman” is needed to trick out ChessBase.

Option 1 with InBetween:

InBetween.exe (ver. 1.5) by O.Gunnar Malin?(download) as your 'UCI engine' in, for example, Chessbase.?InBetween is a Windows command line utility written by Odd Gunnar Malin that can be used to translate or modify text commands between client and server, for instance between a chess GUI (client) and CECP or UCI compliant chess engine (server) to fix communication incompatibilities or glitches in protocol implementations. Further, to track down possible incompatibilities, or for debug purpose, InBetween allows logging of all the communication between GUI and engine (client, server or both), either as output to the console or optionally written into a log file. Along with PuTTY, InBetween can be used for fusing a Windows GUI, such as ChessBase to a chess engine on a Linux server via a Secure Shell. And -pretty stunning because it clearly shows how long the chess community is looking for such solutions - the tools were written 2001 - yes, 21years ago is a LONG time in IT. For those of you who like to do a deeper dive into it, you might this additional link to a conversation with Odd Gunnar helpful. If you want to dive deeper into the history of this great tool, here is a a nice article "The Tale of the Lost Wrapper: InBetween by Odd Gunnar Malin"

Typical challenge to run into: a chess GUI will try to control the engine. Sometimes the command sets of both programs are in conflict. It is here you can use this little utility to adjust the command after it is delivered from the client and before it receives on the server.

Before we get into it, one disclaimer: in other systems it's easier to create a shell script that simply calls SSH, so the focus for both options to connect a remote chess engine cluster is on Windows .. and yes, probably the Apple thing :-)

1. InBetween setup:

Build a separate directory with InBetween, such as "RaspberryCluster". In this directory place InBetween.exe and the InBetween.ini file. Rename or copy with another name. Let's say you rename InBetween.exe to ClusterEngine.exe the ini file needs to have the same name, that said ClusterEngine.ini.

Now we need to configure the ini file.

Syntax:

Debug mode is useful to test things out (as always) where color coding is as follows:

Blue - is flow to the server,?Red?- flow to the client and?Green - Control signal

Be aware that Windows add a boost to foreground applications running with priority class set to normal. You may need to adjust this in the control panel/system if you try to get a foreground app and a background app to run with same priority (engine match). You might want to start the gui with setting Boost= false.

However, I recommend keeping the ini file clean in the first place and start with:

[InBetween]

Priority := high

CommandLine := C:\putty\plink.exe -ssh -C -i your.ppk userid@serverIP "./cluster14"

[Client2Server]

isready

[Server2Client]

readyok

where Cluster14 is our script (for reference see part 1).

for a Telnet connection you need to change the command line parameters to:

CommandLine := PLINK.EXE -raw -P 3333 yourClusterMasterIP "./cluster14"

Option 2 with SSHEngine:

As mentioned in part 1, Matt Nohr, developer and chess player "Matt plays Chess" has done a great job to simplify the connection between chess GUI with SSHEngine, a tool you can find on Gitlab. All KUDOs to him please :-). Latest version can be found here.

I don't need to repeat basic things said under Option 1 or in part 1. SSHEngine is straight forward:

• Use standard SSH to connect to a remote server that has stockfish (or other) engine on it
• Have a configuration file (similar to InBetween) so I can change the connection options easily, especially the host address
• Do not need to use PuTTY or convert my standard SSH keys
• Be able to run the program (from different directories) for different connections

After downloading SSHEngine unzip and move to a folder of your choice such as "EngingeCluster" and create a config file engine.yml in the same directory with the following entries.

If you want to enable extra logging to check what is send etc, add logFileName: "engine.log" to the configuration file.

The log creates on output like the above (from a ChessBase session). And yes, ready to go. Just in case, below a couple of common error messages from Matt's Gitlab page.

Connecting the first GUI - Nibbler, LucasChess and Arena

Not much to say. Follow Nibbler and Arena instructions, start the program, the mpi scripts will be executed and the engine is ready.

You can see after some seconds the engine sends the UCIOK signal and we see our cluster master and working nodes. To crosscheck if everything works well, I added "--display-allocation --hostfile" to the command line. Can be removed when everything went well.

With LucasChess you can add other engines with the only limitation that they use the UCI protocol. The game can be set, limiting the depth of analysis of the motor or the time used to think, or by modifying the way in which it decides. You can also choose the opening, or start in a certain position, or that the engine uses a book of openings or more or less aid. To configure LucasChess is as easy as with Nibbler. Click "New" and choose your "Inbetween" exe file or SSHEngine. LucasChess immediately communicate with your cluster and you can use the GUI to train or play against your cluster.

Connecting ChessBase

If you made it that far, it's just a little step to connect ChessBase with some extra topics to consider. First step is to "create" the UCI engine with a click and the "UCI menu" pops up.

maneuver to your directory with InBetween or SSHEngine. Chessbase will check in the background and then show up if UCIOK comes back from your engine cluster. If so, you see the engine author and engine in the menu.

In this case Stockfish 181022. It's not possible to change that name because of copyright reasons. You might need to play around with Priority and uncheck if a connection problem occurs.

If you click on Parameter ChessBase again communicate with the cluster and the engine parameters are shown. A clear signal everything is ok. I recommend keeping it as is. After "ok" you are back to your standard menu and can add "Kibitzer" aka chess engine.

Do not start the engine now with "Ok", but go first to "Advanced".

And reduce the Hash Table size to 512, you might be able to increase later to 1024 which seems to be the default. And I have no idea why to do this that way, but endless hours of testing have shown this is the way to get the GUI work with the cluster engine. Another important step is to uncheck "Smart CPU" usage. Otherwise ChessBase will overrule the cluster and work with the local amount of CPU.

Now you can see your cluster engine at work. Ignore the amount of CPU (in my case 8 CPU). Reality is that ChessBase not overrule the compute cluster configuration if you uncheck Smart CPU.

Disclaimer: this might not work in the first place, and you might need to play around with Hash Table size, delete the engine configuration and do it again. But it's worth it :-)

Well, this was it for today! Part 3 of this blog will be about Azure Compute Cluster and a Stockfish Cluster Engine. I'm also planning a part 4 to discuss and give guidance on performance optimization. For now, I can lift the curtain and let you know my local raspberry/Le Potato cluster - without overclocking- reach ~ 1.5M nodes per second, up to ~2M nodes per second and the Azure Cluster (3x Standard_D8as_v4) ~ 20M nodes per second, up to ~25M nodes per second

Enjoy and see you soon

Egbert