[parallel postprocess] array operations

Palabos General
#1

Hi!

(1) Would please explain a little about how parallelization is achieved in OpenLB?
(2) I need to do heavy postprocessing like spacial averaging of 1st, 2nd, 3rd moment of velocity (single point correlations) of 3D arrays. If this is done after the collideAndStream() call, will that be executed in parallel? Or I have to implement that as a member function of DataAnalysisBase3D<T,DESCRIPTOR>? Thanks.

Best regards,
Marwan.

#2

This is needed when one attempts to compute turbulent flows using millions of nodes in OpenLB. Question (1) in another form asks: at which level of the code (using which classes) one should add his/her modications so that it works in parallel without having to write MPI statements. This is especially important when using the resulting velocity field to do some extra calculations. Question (2) is an example of an application where (1) is relevant. If after retrieving the velocity field as a 3D array in the top level of the solver, I added a part to the code that calculated covariances of velocity and pressure will that run in parallel? Please feel free to ask any questions in case my description is not clear. Thanks.
Best regards,
Marwan.

#3

Currently, OpenLB offers support only for averages on a scalar quantity. If you would like to know < u[sub]x[/sub]>, then the following command does the trick:


computeAverage(lattice.getDataAnalysis().getVelocity().extractComponent(0)) ;

For quantities like <u[sub]x[/sub] u[sub]y[/sub]> there’s currently nothing you can do, or at least nothing easy. I have added this to the OpenLB wish list, though, and hope it is part of one of the upcoming releases.

An exception to the rule is the velocity norm, for which a predefined function exists:


lattice.getDataAnalysis().getVelocityNorm();

Now, although there’s no support yet for computing these quantities in parallel, you can, in the mean time, still process them sequentially. Be careful to first copy the data to one processor, and then compute the reduction. The following code for example computes <u[sub]x[/sub] u[sub]y[/sub]>. I haven’t tried to compile it, but it helps you get the idea:


TensorFieldBase3D<T,3> const& parallelVelocity = lattice.getDataAnalysis().getVelocity();
TensorField3D<T,3> serialVelocity(nx,ny,nz);
if (singleton::mpi().isMainProcessor()) {
    serialVelocity.construct(); // allocate memory on main processor only
}
copyDataBlock(parallelVelocity, serialVelocity);
double average = 0.;
if (singleton::mpi().isMainProcessor()) {
   for (int iX=0; iX<nx; ++iX) {
     for (int iY=0; iY<ny; ++iY) {
       for (int iZ=0; iZ<nz; ++iZ) {
         average += serialVelocity.get(iX,iY,iZ)[0] * serialVelocity.get(iX,iY,iZ)[1];
       } 
    }
  }
  average /= (double)(nx*ny*nz);
}
cout << "<ux uy> = " << average << endl;