Description of our 3D MHD Computations

    Our final prediction was run on a 3D mesh with 221 x 221 x 432 mesh points (about 21 million cells) in spherical coordinates (r,θ,φ).  Various versions of this run with different parameters were run on two supercomputers, Ranger, a massively parallel supercomputer at the Texas Advanced Computing Center (TACC), and Pleiades, the new massively parallel supercomputer at NASA's Advanced Supercomputing Divison (NAS).

   Pleiades is a high performance computer with 51,200 CPUs and a theoretical peak speed of 609 teraflops/s.  In July 2009 it was ranked as the fourth fastest supercomputer in the world.  Pleiades is an SGI Altix that uses quad-core Intel Xeon E5472 (Harpertown) 3.0 GHz processors and Intel Xeon X5355 (Clovertown) 2.66 GHz processors, with an Infiniband interconnect.  The superb system run by NAS allowed us to run long jobs with minimal queue wait times, allowing us to successfully run our code under the real-time constrains required for a prediction.

   Ranger is a high performance computer with 62,976 CPUs and a theoretical peak speed of 580 teraflops/s.  In July 2009 it was ranked as the eighth fastest supercomputer in the world.  Ranger is a SunBlade Linux Cluster that uses quad-core AMD Opteron (Barcelona) 2.3 GHz processors with an Infiniband interconnect.  The staff at TACC graciously agreed to give us dedicated time to run our simulations without interruption in order to make a timely prediction for the eclipse, and were very helpful in making this happen.

    We used our spherical 3D (magnetohydrodynamic) MHD code MAS, which integrates the MHD equations using semi-implicit (Alfvén and sound waves), fully implicit (diffusive terms), and explicit (flow terms) schemes.  We solve the very large sparse matrix equations generated by these algorithms using a preconditioned iterative conjugate gradient solver.  We set as a boundary condition the radial component of the magnetic field at the base of the corona.  This field is deduced from MDI magnetograms aboard the SOHO spacecraft, which measure the line of sight component of the photospheric magnetic field from space.  Our code is written in Fortran 90 and uses the Message Passing Interface (MPI) for interprocessor communication.  Our code scales very well on many high-performance computer systems.  We have shown essentially linear scaling with processor number up to about 4096 processors.

    Our calculation ran for about 85,000 time steps, relaxing the corona in time (for about 1.8 days of solar time) towards a steady state, thereby approximating the state of the solar corona.  Our time step in the computation was about 1.9 seconds.  The final eclipse prediction runs used 4992 processors on Ranger and 2000 processors on Pleiades, and ran continuously for about 4 days.  We are very grateful for the assistance provided to us by the dedicated staff at TACC and NAS.  Our prediction would not have been possible without these resources.