Genetic optimization with Dakota 3/3: Practice

Part 1: The opti­miza­tion loop
Part 2: A sand­box for exper­i­men­ta­tion
Part 3: Practice

In my first three months of using Dako­ta, I have learned sev­er­al things worth dis­cussing. The con­fig­u­ra­tion described in part 1 of this series makes for a cor­rect­ly-work­ing but crude opti­miza­tion. Each opti­miza­tion is dif­fer­ent, and some exper­i­men­ta­tion and tweak­ing is nec­es­sary for the opti­miz­er to progress efficiently.

I run Dako­ta in a con­text where the eval­u­a­tions are extreme­ly expen­sive com­pared to the oper­a­tion of the opti­miz­er itself. In this con­text, here are notes regard­ing a few topics:

Controlling the size of evaluation group

In the lim­it, hav­ing a very small num­ber of indi­vid­u­als sent for eval­u­a­tion in any giv­en gen­er­a­tion max­i­mizes the effec­tive­ness of the genet­ic algo­rithm, because the active pop­u­la­tion is moved more often towards the Pare­to front, elim­i­nat­ing each time the weak­est mem­bers. If only one new indi­vid­ual is cre­at­ed for each gen­er­a­tion, then by the 100th gen­er­a­tion, this indi­vid­ual is cre­at­ed out of a much high­er-qual­i­ty base data than if it is the 100th new indi­vid­ual in a large first generation.

On the oth­er hand, com­pu­ta­tion­al­ly-inten­sive eval­u­a­tions take time, and it may not be pos­si­ble to com­plete the opti­miza­tion at all unless large num­bers of eval­u­a­tions are ran in par­al­lel. This is an incen­tive to increase the num­ber of new indi­vid­u­als cre­at­ed in each gen­er­a­tion. In those sit­u­a­tions where (human) time is the lim­it­ing fac­tor, the avail­able par­al­lel eval­u­a­tion broad­band becomes the desired num­ber of new cas­es per generation.

Oth­er con­straints may exist, for exam­ple if the machine car­ry­ing out the eval­u­a­tions is not cor­rect­ly admin­is­tered, with local wait­ing queue dynam­ics to be tak­en advan­tage of if the wait­ing time is to be minimized.

In Dako­ta, the num­ber of new cas­es cre­at­ed each gen­er­a­tion is very hard to con­trol. In my (short) expe­ri­ence, it has always grown or decayed expo­nen­tial­ly.
This could be mit­i­gat­ed with tight con­trol over the pop­u­la­tion size in the selec­tion process. How­ev­er, in some opti­miza­tions, the per­for­mance domain is very wide, and there is a need to weed out cat­a­stroph­i­cal­ly bad indi­vid­u­als. This abil­i­ty to remove poor­ly per­form­ing indi­vid­u­als is espe­cial­ly impor­tant since the muta­tion and crossover oper­a­tors are both blind to per­for­mance: they select par­ent indi­vid­u­als indiscriminately.

Ulti­mate­ly, I main­tained some con­trol over the size of the eval­u­a­tion group by inter­rupt­ing opti­miza­tions and re-start­ing them with select­ed pop­u­la­tions. I wish some func­tion­al­i­ty exist­ed that would set upper and low­er lim­its on the num­ber of off­spring in any generation.

Generation-dependent settings

To my knowl­edge, all set­tings passed to moga are fixed for the com­plete opti­miza­tion. This is regret­table, because it makes sense to pro­gres­sive­ly decrease the occur­rence of muta­tion, or the mag­ni­tude of crossover aver­ag­ing, as the opti­miza­tion pro­gress­es towards the final Pare­to front. A workaround is to inter­rupt the opti­miza­tion, save its out­put, and use it as the ini­tial pop­u­la­tion of a new opti­miza­tion with more mod­er­ate settings.

Handling distant individuals

In my under­stand­ing, the crossover mod­ule of moga picks par­ents ran­dom­ly (indis­crim­i­nate­ly) when build­ing groups. In opti­miza­tions where the Pare­to front ranges over a very wide range of para­me­ters, mem­bers that are very dis­tant from one anoth­er may not be use­ful­ly crossed. This unpro­duc­tive cross­ing (pro­duc­ing low-per­form­ing mem­bers) becomes more fre­quent as one pro­gress­es over the opti­miza­tion, since the Pare­to front becomes longer and thinner. 

I do not see an imme­di­ate solu­tion to this issue. Again, a workaround is inter­rupt the opti­miza­tion, “split” the domain of para­me­ters into sev­er­al sub­do­mains, and assign one new opti­miza­tion to each. How­ev­er, this like­ly tends to result in slow­er con­ver­gence and a more dis­con­tin­u­ous Pare­to front, since the sub­do­main opti­miz­ers are not com­mu­ni­cat­ing with one another.

Using remote evaluations

Dako­ta com­mu­ni­cates with exter­nal eval­u­a­tors through the file sys­tem, await­ing a response file for each eval­u­a­tion, in which it reads the out­put. The same tech­nique can be used to run eval­u­a­tions on remote machines.

I con­fig­ured Dako­ta to run on my local desk­top. The eval­u­a­tor is a shell script which, run­ning from each local eval­u­a­tion fold­er, successively:

• assem­bles and con­cate­nates con­fig­u­ra­tion files locally;
• syn­chro­nizes files with a remote high-per­for­mance clus­ter, using rsync;
• sub­mits a slurm job for the com­pu­ta­tion to be car­ried out on the clus­ter. That com­pu­ta­tion is con­fig­ured to cre­ate a file “finished” in the clus­ter work fold­er once it has completed;
• sleeps a num­ber of hours (to avoid congestion);
• checks for the exis­tence of the “finished” file on the remote cluster;
• retrieves key files from the clus­ter to the local machine;
• per­forms post­pro­cess­ing with a local Python script, which itself out­puts the pre­cious response file which is await­ed by Dakota.

In this way, the oper­a­tion is car­ried with the best of both worlds.
The opti­miza­tion prop­er takes place local­ly, where there are no time con­straints, and where instal­la­tion, con­fig­u­ra­tion and admin­is­tra­tion are eas­i­ly car­ried out. The com­pu­ta­tions them­selves are car­ried out remote­ly on a clus­ter, and sub­mit­ted in small batch­es on a rolling basis, which min­i­mizes total wait­ing time.

Handling gaps in population

As not­ed above already, moga selects par­ent indi­vid­u­als indis­crim­i­nate­ly (ran­dom­ly). It does not attempt to com­pen­sate for thin­ly-pop­u­lat­ed areas of the para­me­ter domain. The indis­crim­i­nate sam­pling tends to rein­force already-dense areas, and there is a risk that parts of the domain remain unex­plored. This is espe­cial­ly true in the edges of the domain, because while muta­tion can in prin­ci­ple cre­ate any kind of off­spring, crossover can only pro­duce off­spring which stand “in between their par­ents” (their input para­me­ters are always with­in the range of those of their parents). 

I do not know of a solu­tion to this issue (although a workaround is again to inter­rupt the opti­miza­tion and ini­tial­ize the fol­low­ing opti­miza­tions more care­ful­ly). This self-rein­force­ment of well-pop­u­lat­ed domains is a fun­da­men­tal fea­ture of genet­ic algo­rithms. Dako­ta sup­ports hybrid approach­es, in which one may run a genet­ic opti­miz­er, fol­lowed by anoth­er method such as a gra­di­ent-based search; I have not yet looked into this functionality. 

Specifying constraints

I have not yet learned how to spec­i­fy con­straints between input vari­ables in Dakota. 

Moga does not attempt to know why an indi­vid­ual per­forms well or poor­ly, and so does not fun­da­men­tal­ly dis­tin­guish between a poor­ly-per­form­ing and an invalid indi­vid­ual. In this view, one could think (as I did) that the rejec­tion of invalid indi­vid­u­als may just as well be car­ried out by the eval­u­a­tor (where pro­gram­ming of con­straints may be more con­ve­nient) instead of by Dako­ta itself. This approach works, but has two unde­sir­able consequences:

1. This fur­ther reduces the con­trol over the num­ber of new indi­vid­u­als to be sent for eval­u­a­tion (Dako­ta nev­er knows how many of the off­spring it pro­duces are invalid);
2. When the domain of inputs is not uni­form, this bias­es Dako­ta against “nar­row” domain areas. For exam­ple, if in one end of the input para­me­ter domain, many com­bi­na­tions of domain para­me­ters are invalid (not per­mit­ted by con­straints), then the prob­a­bil­i­ty that Dako­ta will pro­duce invalid indi­vid­u­als is high­er in that area. Con­se­quent­ly, “nar­row” domain areas receive pro­por­tion­ate­ly less chil­dren, and become less pop­u­lat­ed, exac­er­bat­ing the pop­u­la­tion gap prob­lem described above.

For that rea­son, if the con­straints on the input para­me­ters do not apply uni­form­ly on the input design space, it is well worth your time to spec­i­fy them in Dako­ta itself rather than in the evaluator.

Preventing repetition

In sev­er­al of the issues dis­cussed above, one workaround was to restart new opti­miza­tions with mod­i­fied set­tings or scope. In cas­es where the eval­u­a­tions are com­par­a­tive­ly com­pu­ta­tion­al­ly expen­sive, it is often worth restart­ing an opti­miza­tion with more ade­quate set­tings rather than risk wast­ing com­pu­ta­tion­al resources. When doing so, it is impor­tant that no eval­u­a­tion car­ried out pre­vi­ous­ly is ever repeated.

Ide­al­ly, a data­base could be fed to Dako­ta, in which it would search through pre­vi­ous­ly-ran cas­es before launch­ing any eval­u­a­tions. I do not know of a pos­si­bil­i­ty to do this, and I imple­ment­ed this at the eval­u­a­tor lev­el instead. The first step of my eval­u­a­tor script is to parse a csv file (the con­cate­na­tion of all for­mer opti­miza­tion tabular_data_file sum­maries) and check whether an entry match­es the cur­rent eval­u­a­tion. If so, a response file is imme­di­ate­ly cre­at­ed and the eval­u­a­tion ter­mi­nates before any com­pu­ta­tion is car­ried out.

Gen­er­al­ly, I found that hav­ing a clear, well-script­ed roun­tine for rapid­ly deploy­ing new opti­miza­tions is a major advantage.

Conclusion

Genet­ic opti­miza­tion with Dako­ta is con­cep­tu­al­ly not com­pli­cat­ed, but a lot of thought must go into the set­up and mon­i­tor­ing of the opti­miza­tions if mean­ing­ful results are to be obtained in rea­son­able amounts of time. I shared here every­thing I learned over the last three months in the hope it can help oth­ers. I hope oth­ers may sim­i­lar­ly share their expe­ri­ences online.

Part 1: The opti­miza­tion loop
Part 2: A sand­box for exper­i­men­ta­tion
Part 3: Practice