Interview with Don Dailey (USA)
Programmer of RexChess, Socrates, CorelChess, Mini, CilkChess, actual "Doch"
18th - 20th, December 2009
by Don Dailey, Frank Quisinsky, members of TalkChess
Don Dailey (USA)
Unexpectedly and surprisingly ... middle of November 2009 a new engine by Don Dailey was free available. Don Dailey was one of the fathers of computer chess. His latest project CilkChess was never commercial or freely available. CilkChess was a mainframe and Don plays with CilkChess in different computer chess events only. Nobody knows how strong the playing strength of CilkChess is to buy at normal computers everybody can use. More or less CilkChess, perhaps Ferret (also an American engine) and ChessWizard (French engine) are the ghosts of the best available commercial products in these times, so like Shredder or Fritz for examples.
Reason enough to find out more about Don Dailey and his new engine Doch ...
great that you find the time for this interview.
I am sure that many of your fans will read it.
I am aware of this, that thousands of computer chess fans spent in MS-DOS times millions of hours playing with RexChess. RexChess 2.1 / 2.3 = CULT. Do you remember about it and the "TimeRef" you had programmed for RexChess. Perhaps the first chess benchmark which was directly switched from one computer chess fan to the next. A nice time, RexChess was hazardous material of depending. RexChess impressed by its graphic as well as performance in the past. You had a lot of success in Germany! In America too? Perhaps you can tells all a little story about experiences in this time.
In RexChess we came up with some ideas that we thought were original but later discovered others were doing also. I am not sure if others were doing them when we were, but I suspect they were. We had never heard of null move pruning, but invented something of our own that works on similar principles based on a static analysis of what was threatened and whether it could be ignored or not based on what was going on with the alpha/beta window.
The TimeRef feature no longer works for RexChess on fast computers because of a "divide by zero" error! When we designed this feature we did not anticipate how fast computers would get in such a short period of time. The timing loop that we used to estimate the speed of the computer operates in less time than the resolution of MSDOS machines could handle and so it came out to zero! Thus RexChess now will crash if run on a modern computer.
I remember than Larry and I worked together in a very intense way for something like a 2 week period to gain some big advances before we released it. By todays standards the graphics and the strength of RexChess would be embarrassing, but it at the time it did receive a lot of praise.
RexChess was an assembly language program with the graphics and user interface in Borland's Turbo Pascal.
In the late '80s and early '90s after RexChess Socrates was coming out. A better graphical user interface and the first engine with knowledge in positional middle games - think so -. Which parts of chess programming are most important for you in these times?
At the time, it was considered very important to do well on problem sets. So we payed more attention to that. Of course today we know that is not as important as we thought back then, because performance on tactical problem sets is only partially correlated with playing strength. But we had more tactical extensions and focused more on evaluation too because some sets at that time were also positional. Of course Larry Kaufman also had published problem sets that we wanted to do well on. Eventually we learned that we could optimize a program to do well on a particular problem set, without increasing it's actual strength very much - but we assumed that it would at least help some.
Even though the end user could not see this, I had created something I called a "rulebase", which was a way for Larry Kaufman to plug knowledge into the program without my help. I had created a little language that provided a way for him to add knowledge in a more sophisticated way than simply adjusting weights.
It was also during those days that we started learning the important of massive testing. In the RexChess days we TRIED to test but we did not have the resources to test like we do now. I remember that between us we had 5 or 6 PC XT machines (at something like 4 or 5 MHZ) and we would run for 24 hours just to get 100 or 200 games. I cannot remember the exact numbers but it was pathetic!
I also invented a universal auto-tester. In those days there was no winboard, or UCI or any standardized interface and we wanted to test against the good programs of the time. And we did not want to do it manually. Remember this was still in the MSDOS days. So I build a tester that worked with 2 computers and ANY chess program could be configured to work with it. It worked by installing a "memory resident program" which periodically looked at the screen to notice if any pixels had changed! When it detected a change it was decoded and the appropriate keyboard presses were faked. From the chess programs point of view someone was typing moves from the keyboard! Each program had to be configured with relevant information such as how big each square was and which pixel to look at for state changes, but it actually worked very well.
We can read that GM Larry Kaufmann is involved in most of the developments. What is exactly the work of GM Larry Kaufmann? By my information Larry created opening books for Novag chess computers, perhaps for you too, as your partner? Or did give Larry helpings in programming? In this case ... for which parts of programming? The question is interesting because Larry is member of the Rybka team today.
Larry is a very important part of the team and we work together extremely well. We have a certain chemistry. I have tried working with other strong players in the past, but it takes a special kind of pragmatism which Larry possess. There are extremely strong players for example who do not have the ability to clearly articulate things in a way that can be translated directly to a chess program.
Larry does not do any of the actual programming, but he understands almost as well as I do how chess programs work. So he is more than just the "evaluation guy." A lot of what appears in our programs are the combined result of much discussion between us. We bounce ideas off of each other and come to some agreement about whether "we" should try to implement it or not. This includes even search heuristics and things not directly related to evaluation.
After realising Socrates coming out a "Big Store" product called "CorelChess". And after CorelChess a big break by yourself? Do you lost your interest in chess or do you have other reasons for your longer break.
Cirmunstance mostly. Took a job at a startup company, got married, etc.
In the late '90s CilkChess gave more furore. Is CilkChess a privat mainframe project? Why do you give up CilkChess and which experience do you collect? What is important for you in these times? Why do you never release a CilkChess version for everyone as commercial or free product?
CilkChess was designed only to run on parallel machines and no consideration was given for it to be a strong PC program and this showed. Even though it could be compiled to run on any Unix system, it did not play that strongly on an ordinary PC and was optimized only for DEC's alpha processor. The program basically blew out the memory cache of ordinary PC's and performed poorly on them. Not to mentioned the fact that it was a 64 bit program and it really needed to be run that way.
But on the alpha it was a totally different animal. One version of that program won the 11 round Dutch Computer Chess Championship wihout a single loss and only 2 draws.
I would like to mention that CilkChess was developed at MIT with the help of many students. But it did not have much of Larry Kaufman's influence and I think it could have been much stronger if it had. On the other hand there were students at MIT who contributed and definitely helped improve the program - but I don't think it made up for the lack of Larry!
Let us speak about Doch. In German language "doch" is an expletive word / filler word and it looks very strangely as a name of a chess program. What did you mean with "Doch"? Let me give you a little hint ... renames your engine in Achilles :-) After the "Trojan" interview (by Alexander Schmidt and myself) with Stefan Meyer-Kahlen (Shredder, Germany) Stefan don't want to rename Shredder in Achilles. All what I mean is, imperatively all we need is an Achilles in Rybka-Times!
Doch was never intended to be the name that would stick. When I first decided to write this program I needed a name and did not want to spend days obsessing over it. I did not want to call it "chess" but it needed a name to give it some personality. Doch stands for DOns CHess. I never got around to giving it a proper name and I feel a bit immodest calling it after my own name!
How long do you work on Doch before you released the first version? Why your first available version of Doch is so strong and can play directly under the TOP 10 in the world (today more as 400 engines are available). Perhaps parts of your CilkChess sources were included? I am sure you will give us hints about your secrets.
It's hard to say because I did 2 or 3 programs and threw them out and started over again. I am a kind of perfectionist and it's important to get the data structure right because you will have to live with it forever. I was reading Ed Shroeders articles on how Rebel works and I was intrigued, so I tried to write a program like that, but eventually decided that 64 bits is the way to go. It sounds like a horrible waste of time, but it trains you to understand how things work and what the compromises are, and there always compromises.
Even Doch has had large sections of code scrapped and re-written.
Larry Kaufman and I had been out of touch for a few years and I had already written an early version of Doch when Larry got interested in helping me. It started very slowly, a tiny bit of advice here and there, and gradually we were working together again.
Which parts of chess programming today are most interesting for you? Do you have a look in free sources like Glaurung / Stockfish, Fruit, Crafty, or others? Do you think that actually ideas can give us chess engines with 3.200 or more ELO in the near future?
There is no good program that is not based on the ideas that have been passed along over the years. We all stand on the shoulders of giants. I make it a point to at least give a quick look at the source code of any open source chess program I can find that is reasonably strong. A lot of very smart people have contributed good ideas that we all use.
One of the surprises of computer chess is not hardware, but the fantastic pace of software development. Years ago it seemed almost impossible to make my program much stronger, but if I had known then what I know today, I would have been very surprised. So almost nothing would surprise me about the possible improvements for the future.
One of the main topics for users of chess engines are w32 / x64 and multiprocessing. I think that max. 20 - 30% of the community are using x64 operating systems. Do you optimize Doch for w32 or x64 systems? Will be a multiprocessing version available in the near future?
I optimize for 64 bit processors - doch is a 64 bit program. I do plan to eventually have a multiprocessing version and Doch was designed so that this would be easy.
In just 2 or 3 years most people will be using 64 bit operating systems.
Doch is your first free available engine. Do you have commercial intention about the future of Doch? Or do you think that today it isn't possible to make money in chess programming?
That is an open question. I would prefer making a living writing chess programs (who wouldn't?) and if I were payed for it I could do a better job because I could spend much more time doing it. It is hard to compete with people who do this full time. My guess is that it is pretty difficult to make money in computer chess unless you have one of the top 2 or 3 programs.
It seems that Doch have problems in endgames. By your information Doch isn't finished. Perhaps one reason for your statement would be the endgame? Easy to see that Doch is playing fantastic computer chess in the middle game. Clear for me, you don't like endgames! Right?!
I agree that the endgame is weaker than the middlegame. Larry and I will probably turn our attention to the endgame soon. It's difficult to say how much of a factor it is but as Doch gets stronger, the endgame weaknesses will become more and more of an "achilles heel."
Private question but I will release your answer too!
Most of computer chess pioneers are Americans, like John Stanback, teacher and programmer Prof. Robert Hyatt, Bruce Morland, Don Dailey -yourself-. Possible that it's only a chance? Do this group of persons switch their information through all the years? Is this group of persons a friendly club with regular "gab sessions". Or is there a little competition among each other.
We are all friends and have actively shared our ideas. Bob Hyatt and John Stanback are VERY OPEN to sharing ideas to this day. Bruce Moreland too. But we don't have a special club or anything like that. In the past I have bounced a lot of ideas off of John Stanback.
Over the years some have been much more secretive than others. For instance Richard Lang does not give aways secrets, but Bob Hyatt not only gives his sources freely, but will explain them to you if you ask. And of course most of the commercial guys won't tell you anything. I am somewhere between. I do have some secrets that I have not shared - I still want to have at least a tiny advantage!
Don Dailey answering questions ...
Questions by others ...
I asked in TalkChess for interview questions, means that others should have the possibility to ask some things too.
Why is it named Doch?
09-980 number - sounds like many versions. So how long has it been in the making?
980 version have been produced in 2009 at that version number. Actually more than that because I started using that versioning system in the middle of the year. I am now up to 1200+ for this year. Most versions are very minor changes though.
Any chess programs that inspired you?
They all do, each program is like a person with a personality. Richard Lang's programs always amazed me, even though many people were critical of it's playing style. I don't care if a program plays boring or exciting, the main thing is whether it can win. I always found it odd that people would criticize a program for winning as if it were somehow not playing fair.
Of the modern programs I really like glaurung/stockfish.
Socrates connection - any remnants from it?
No, it is dead!
Future projections on other releases?
Will you go commercial?
Will you do a commercial ? - just kidding
Future endgame bases, multi processor?
Don't really know yet which direction we will go.
Gian-Carlo Pascutto (programmer of Sjeng)
You have been working in the games programming community for a long time. When you, with your experience, made a new chess engine, what did you do differently from before and what do you focus on now?
More and more emphasis on the engineering aspects now. It's my opinion that you cannot compete without feedback from massive testing - not just massive but very careful testing. It's very easy to think you have an improvement but you have regressed. But I think there is still huge room for creativity.
One thing of course is that you have to keep up with new modern ideas. The best programs of today do evaluation better than many years ago. One big surprise for me has been how much of Doch's improvement has been evaluation related.
Most people who have been in computer chess for a long time know that my programs have been weak in king safety. This is one area I was determined to improve on this time around and Doch is by far the best program I have done in this regard. But still I think it can be improved a lot with even better king safety.
Gian-Carlo Pascutto (programmer of Sjeng)
You have written engines for other games. Are there techniques, ideas or viewpoints you find interesting and want to try in the chess engine?
I wonder if MCTS (Monte Carlo Tree Search) as used in the newer Go programs could be effectively adapted to Chess. It will be difficult to improve on modern computer chess search techniques, but I would not mind seeing some effort in this area or even looking at it myself. I did write a MCTS Go program but I know almost nothing about Go so I feel handicapped in that regard. But it might be worth a good look with Chess. This is something you could also persue being both a great chess programmer AND a very good Go programmer.
Where do you see the computer chess field in 5 years? 10 years?
The convential wisdom is that computers are going to have more cores but the speed will advance more slowly. So one fruitful area to explore is how to take advantage of this trend.
I think very shortly we are going to see a big jump in the strength of the top chess programs since the recent reverse engineering of the mighty Rybka program - which exposes more good ideas to the world.
When the source code of a strong chess program becomes available, someone will find a way to improve it, even if they could not have written such a strong program by themselves. It still takes a good engineer to be able to do that, but it is what always happens. But whether you simply work on someone elses code or you just borrow those ideas for your own chess program, the end result is the same - you get a stronger chess program. In one form or another this is what has happened over the years. Ideas are exchanged either by word of mouth, or directly from source code or in published articles.
In 10 years we are going to see some pretty impressive chess playing software/hardware combinations.
Any guess as to how long until the top programs draw virtually all games against each other?
A very long time. My feeling is that chess programs still have a few hundred ELO of improvement left before they become omniscient.
Opening books: What role has it played in computer chess development and what future role do you see it playing?
They will always be important and perhaps even more important as program get stronger and stronger. Computer checkers is a good example of game that is closed to being solved by PC programs and yet opening books still play a major role.
When do you expect chess will be solved? To be more specific, do you expect there to be a formula where a positional can be plugged in that would output win, draw, or loss?
If you do the calculations, it would seem that such a thing is impossible from the practical point of view. But it's funny how things that once seemed impossible seem to become possible when you attack the problem from all angles.
At MIT I wrote a Rubiks cube solver which finds the minimal solution. At the time the program took about 2 hours to produce a solution but i suspect my quad core machine could do it in 5 minutes or something like that. When I first started building this program it seemed like an overwhelming and impossible thing. I kept finding clever tricks to make this faster and faster and all of a sudden it was possible, then it was a reality. This was a joint effort with Keith Randal and others at MIT.
Have thanks for your time Don.
SCHACHWELT wish you and your family Merry Christmas and a lot of SUCCESS in your new development of Doch !!!
Thanks to the members of TalkChess for the additional questions!
Copyrights and Sources by http://www.schach-welt.de & Frank Quisinsky
Further publication of this interview is allowed and wished,
but with the permission of http://www.schach-welt.de only.
One condition is the complete casting (e.g. inprinted media).
Frank Quisinsky, December 20th, 2009