Hi everyone!
I am trying to run a simulation on the cluster at my university. For the cluster i have to specify the number of cores the simulation will run on and the total amount of allocated memory. If the simulation needs more memory than allocated the cluster will kill my simulation.
For the calculation my simulation needs 20GB in RAM. Everytime i write a VTK (3GB) output file or save the state of the simulation (8GB) the memory is filled up.
Example:
t=0, 20GB in RAM
t=100, 20+8+3 = 31GB in RAM
t=200, 20+8+3+8+3 = 42GB in RAM
and so on…
Now when the needed memory hits the maximum allowed amount, the cluster kills my simulation.
On my local computer (32GB total RAM) this problem also occurs (the memory is filled up completely to 32GB) but the simulation is still running because the new 8+3GB are overriding the old 8+3GB in memory. I know this because i have the “system monitor” installed (ubuntu 14.02LTS) and in the upper right corner of my monitor i can see the memory usage. The 20GB of the simulation RAM are displayed in a dark green colour and the additional 8+3GB in bright green colour. It seems to me that on the cluster this override does not happen and it therefore kills the simulation.
Do you know how to stop palabos from keeping VTK and checkpoint data in the memory? Here are some snippets of my code:
runProgram(){
MultiBlockLattice3D<T, DESCRIPTOR> *lattice;
MultiScalarField3D<T> *rhoBar;
MultiTensorField3D<T, 3> *j;
std::vector<MultiBlock3D*> lattice_rho_bar_j_arg;
[…]
// Initialize Lattice
std::vector<MultiBlock3D*> checkpointBlocks;
checkpointBlocks.push_back(lattice);
checkpointBlocks.push_back(rhoBar);
checkpointBlocks.push_back(j);
[…]
// run simulation
pcout << "Starting simulation." << std::endl;
for (plint i = iniIter; i < param.maxIter; ++i) {
[....]
writeVTK(lattice, [...]);
if ((i % saveIter == 0 && i != iniIter)) {
pcout << "Saving the state of the simulation at iteration: " << i << std::endl;
saveState(checkpointBlocks, i, true, "continue.xml", "checkpoint_", 8);
pcout << std::endl;
}
lattice->executeInternalProcessors();
lattice->incrementTime();
}
The function “writeVTK” looks as follows:
void writeVTK(MultiBlockLattice3D<T, DESCRIPTOR>*& lattice, […])
{
plint nx = lattice->getNx();
plint ny = lattice->getNy();
plint nz = lattice->getNz();
Box3D fullDomain(0, nx - 1, 0, ny - 1, 0, nz - 1);
VtkImageOutput3D<T> vtkOut(createFileName("volume", iT, PADDING));
std::auto_ptr<MultiScalarField3D<T> > p = computeDensity(*lattice, fullDomain);
std::auto_ptr<MultiTensorField3D<T, 3> > v = computeVelocity(*lattice, fullDomain);
vtkOut.writeData<float>(*computeNorm(*v), "velocityNorm", param.dx / param.dt);
vtkOut.writeData<3, float>(*v, "velocity", param.dx / param.dt);
vtkOut.writeData<float>(*p, "pressureLB");
}