I/O problems on BlueGene/P

Palabos General
#1

Dear guys,

I am using PalaBos on BlueGene/P( BGP). PalaBos is powerful and much helpful for my work. However, I am meeting two problems of I/O.

  1. When I use vtkOut.writeData or vtkOut.writeData to output the fields, it works well on my local machine(64bits), but writes wrong fields on BGP(32bits). The numbers of the density or velocity in the VTI files are large abnormally (such as 1.0e+38 for float and 1.0e+308 for double). I am sure this problem is caused by the I/O, as I wrote a function to output the fields in ASCII format and the code works well on BGP.

  2. For saving checkpoint, the saveBinaryBlock can output large file (such as 25G) on BGP. However, when I try to use parallelIO::save, it works. But the size of the file can’t be larger than 2G. Moreover, when I use parallelIO::load to load the checkpoint, it can’t work even with small checkpoint file (such as 100M).

Is there any method to fix these problems? Thanks very much.

Yong

#2

Dear Yong,

Yes, the BlueGene/P has a weird architecture: every node is 32-bit, yet the full machine is used to simulate problems much larger than 2 GB. To get a proper behavior for large problems, you need Palabos to use the 64-bit integer type “long long int” for I/O operations, which you get when you compile it with the PLB_BGP flag (add -DPLB_BGP to compileFlags in the Makefile).

The next thing with the BlueGene is that you really need to use parallel I/O to write/load the checkpoint files, because the I/O bandwidth of a single node is incredibly low. Here’s how to do this in Palabos. Right after the plbInit line at the beginning of your program, write:


global::IOpolicy().activateParallelIO(true);

Then, write your data through something like


parallelIO::save(lattice, "lattice");

(don’t provide an filename extension). You can write any block-lattice, scalar-field or tensor-field with this. The load your stuff through


parallelIO::load("lattice", lattice);

As for the VTK files, this is probably because the BlueGene stores binary data with the big-endian convention, while the PC on which you visualize it is little-endian. Fortunately, the VTK format supports conversion of endianness. In your case, the simplest thing to do is probably to replace the string “LittleEndian” by “BigEndian” in your VTK file, for example with the following call to sed:


sed -i "s/LittleEndian/BigEndian/g" myOutputFile.vti

By the way, if ever you want to read an STL file on the BlueGene, use an ASCII-format STL to avoid issues wrt. endianness.

Cheers,
Jonas

#3

Hi Jonas,

Thanks very much. Your reply is much helpful.

I had read the previous discussion about parallelIO in this forum before I asked my questions. In fact, I did all you said except “don’t provide a filename extension”. I used *.dat as the name of the checkpoint files, and then blocks’ data and layouts were written into one file,and then of course can’t load correctly. Now, it works.

As for binary file, you are right. To replace “LittleEndian” by “BigEndian” is available. I should have thought of this way : ) . About STL file, yes, I had found the difference and used ASCII format.

Thanks again.

Regards,
Yong

#4

Hi,

I used the above method to make palabos run on Blue Gene/P machine successfully.

While another problem bothers me that it take very long time to output the data to vti file.

For example, the size of the simulation is 512512512 and it took about 12 minutes for 3600 iterations with 1024 cores, while it took about 18 minutes to output the velocity and velocity normal to a single vti file.

The output functions used are listed as follows,

     vtkOut.writeData<float>(*computeVelocityNorm(lattice), "velocityNorm", 1.);
     vtkOut.writeData<3,float>(*computeVelocity(lattice), "velocity", 1.);

I want know if it is normal or something wrong happened.

Thank you very much .

Kai

#5

Hello Kai,

There exists no straightforward solution to your problem. On the BlueGene, it is crucial to use parallel I/O to write data, because the writing speed of a single node is excruciatingly slow (much slower than the typical I/O speed of a desktop PC). However, while Palabos uses parallel I/O to write checkpoint files, VTK files are, at the current stage, written with non-parallel speed.

The only recommendation I can provide is that, instead of writing VTK files, you write frequent checkpoint files from which you extract the VTK in a post-processing stage: just write a short Palabos application that reads the checkpoint file, computes some values and writes them into a VTK file. This program can for example be executed on the login node of your BlueGene which has much faster I/O access to your data files than a single computational node.

This is of course not so elegant, and the BlueGene’s head node was probably not intended to be used in this way, but sometimes you just need to get things done in a way or another. If you get caught by your BlueGene’s sysadmin, you can still mumble something about the insufficient I/O speed of the BlueGene computational nodes.

#6

Hi, Jonas,

Thank you for your response.

Yeah, it is true. When you work with some giant machine, you have to some compromise ( or something not so elegant as you said). However, your response is really helpful to me. I need to know nothing wrong happened.

Thank you again.

Best Regards,
Kai

#7

As for the VTK files, this is probably because the BlueGene stores binary data with the big-endian
convention, while the PC on which you visualize it is little-endian. Fortunately, the VTK format
supports conversion of endianness. In your case, the simplest thing to do is probably to replace
the string “LittleEndian” by “BigEndian” in your VTK file, for example with the following call to sed:


sed -i "s/LittleEndian/BigEndian/g" myOutputFile.vti

Hi everyone,

I want to copy the binary checkpoint files from BGP(32-bit) and use them to initialize the flow on some other 64-bit platform. Is there any similar method to convert? Seems we can’t just replace LittleEndian by BigEndian.

Any message is welcome. Thanks.

Yong