SWQU Workshop Installation Instructions
Linux Ubuntu 20.04 or 22.04 based distro
Prerequisites:
- MUST HAVE ACCESS TO INTERNET
- MUST BE ABLE TO INSTALL APT PACKAGES
- Open a Terminal
Install required Ubuntu packages:
sudo apt install bash make build-essential gfortran tar openmpi-bin libopenmpi-dev git libhdf5-dev libhdf5-openmpi-dev python3-pip hdf5-tools ffmpeg htop libblas-dev liblapack-dev qtbase5-dev xterm x11-apps
Install Python Packages:
pip3 install numpy matplotlib h5py scipy zenodo_get pandas astropy sunpy
Setup SWQU Installation
Create the main installation folder:
mkdir swqu_workshop
mkdir swqu_workshop/cpu
cd swqu_workshop/cpu
Create a startup script:
touch load_swqu_cpu
chmod +x load_swqu_cpu
Set number of CPU threads
Edit the startup file load_swqu_cpu
using a text editor (such as nano
or vim
) and insert this line:
export OMP_NUM_THREADS=
[NUMBER_OF_CPU_THREADS]
You can use tools like lscpu
and/or htop
to determine [NUMBER_OF_CPU_THREADS].
Source the startup script with the command:
. load_swqu_cpu
Install Open-source Flux Transport (OFT)
Create the OFT installation folder:
mkdir oft
cd oft
Clone the git repository:
git clone --recursive https://github.com/predsci/oft.git/ .
Enter the HipFT directory and build HipFT:
cd hipft
./build_examples/build_gcc_ubuntu20.04_cpu.sh
Test the HipFT installation:
cd testsuite
./run_test_suite.sh -mpicall='mpirun -bind-to socket -np'
cd ..
Return to the swqu_workshop/cpu
folder:
cd ../../
Install Solar Wind Generator (SWiG)
Create the SWiG installation folder:
mkdir swig
cd swig
Clone the git repository:
git clone --recursive https://github.com/predsci/swig.git/ .
Enter the MapFL directory and build MapFL:
cd mapfl
./build_examples/build_cpu_multithread_gcc_ubuntu20.04.sh
Test the MapFL installation:
cd example
../bin/mapfl mapfl_get_simple_open_field_map.in
If the run is sucessful, there should be a file called ofm.h5
in the directory.
Enter the POT3D directory and build POT3D
cd ../../pot3d
./build_examples/build_cpu_mpi-only_gcc_ubuntu20.04.sh
Test the POT3D installation:
./validate.sh
The validation script will indicate if the run has passed.
Add the following lines to the load_swqu_cpu
startup script:
export PATH="<DIR>/swqu_workshop/cpu/oft/bin:$PATH"
export PATH="<DIR>/swqu_workshop/cpu/oft/hipft/bin:$PATH"
export PATH="<DIR>/swqu_workshop/cpu/swig:$PATH"
export PATH="<DIR>/swqu_workshop/cpu/swig/bin:$PATH"
export PATH="<DIR>/swqu_workshop/cpu/swig/pot3d/bin:$PATH"
export PATH="<DIR>/swqu_workshop/cpu/swig/pot3d/scripts:$PATH"
where <DIR>
is the path to the directory that swqu_workshop
was made in.
NVIDIA GPU Installation (ADVANCED)
Create the main installation folder:
Assuming you are in the directory you started it at the top of these instructions:
mkdir swqu_workshop/gpu
cd swqu_workshop/gpu
Create a startup script:
touch load_swqu_gpu
chmod +x load_swqu_gpu
Install the NVIDIA HPC SDK compiler:
curl https://developer.download.nvidia.com/hpc-sdk/ubuntu/DEB-GPG-KEY-NVIDIA-HPC-SDK | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-hpcsdk-archive-keyring.gpg
echo 'deb [signed-by=/usr/share/keyrings/nvidia-hpcsdk-archive-keyring.gpg] https://developer.download.nvidia.com/hpc-sdk/ubuntu/amd64 /' | sudo tee /etc/apt/sources.list.d/nvhpc.list
sudo apt-get update -y
sudo apt-get install -y nvhpc-24-3
Edit the startup file load_swqu_gpu
using a text editor (such as nano
or vim
) and insert these lines:
#!/bin/bash
version=24.3
NVARCH=`uname -s`_`uname -m`; export NVARCH
NVCOMPILERS=/opt/nvidia/hpc_sdk; export NVCOMPILERS
MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/$version/compilers/man; export MANPATH
PATH=$NVCOMPILERS/$NVARCH/$version/compilers/bin:$PATH; export PATH
export PATH=$NVCOMPILERS/$NVARCH/$version/comm_libs/mpi/bin:$PATH
export MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/$version/comm_libs/mpi/man
. /opt/nvidia/hpc_sdk/Linux_x86_64/${version}/comm_libs/12.3/hpcx/hpcx-2.17.1/hpcx-mt-init-ompi.sh
hpcx_load
# If there are any issues with MPI+CUDA, one can try usiing the old OpenMPI v3.
# Simply comment the above 4 lines and uncomment the following 2.
# Then, re-compile the code and try running again.
#export PATH=$NVCOMPILERS/$NVARCH/$version/comm_libs/openmpi/openmpi-3.1.5/bin:$PATH
#export MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/$version/comm_libs/openmpi/openmpi-3.1.5/man
export CC="pgcc"
export CXX="pgc++"
export FC="pgfortran"
export F90="pgfortran"
export F77="pgfortran"
export CPP=cpp
export OMP_NUM_THREADS=
[NUMBER_OF_CPU_THREADS]
Source the startup script with the command:
. load_swqu_gpu
This should work: nvfortran --version
Compile HDF5 library with NV:
The HDF5 library must be compiled with the same compiler as the codes, so we need to compile from source:
mkdir hdf5
cd hdf5
wget https://docs.hdfgroup.org/archive/support/ftp/HDF5/prev-releases/hdf5-1.8/hdf5-1.8.21/src/hdf5-1.8.21.tar.gz
tar -xf hdf5-1.8.21.tar.gz
cd hdf5-1.8.21
export CFLAGS="$CFLAGS -fPIC"
./configure --enable-fortran --enable-production
make -j <NUM_CPU_THREADS>
make install
mv hdf5/* ../
Add the following line to your load_swqu_gpu
startup script:
export LD_LIBRARY_PATH=
[HDF5_INSTALL_DIR]/lib:${LD_LIBRARY_PATH}
where [HDF5_INSTALL_DIR] is the full path to the directory where the NV HDF5 library is installed (swqu_workshop/gpu/hdf5
)
Source the startup script:
. load_swqu_gpu
Install Open-source Flux Transport (OFT):
Create the OFT installation folder:
mkdir swqu_workshop/gpu/oft
cd swqu_workshop/gpu/oft
Clone the git repository:
git clone --recursive https://github.com/predsci/oft.git/ .
Enter the HipFT directory and build HipFT:
cd hipft
Copy the GPU build script:
cp build_examples/build_nvidia_gpu.sh .
Open build_nvidia_gpu.sh
and set the HDF5 lines to be:
HDF5_INCLUDE_DIR="
[HDF5_INSTALL_DIR]/include"
HDF5_LIB_DIR="
[HDF5_INSTALL_DIR]/lib"
Also, set the correct "cc" in the GPU lines (there are two). See the CUDA wikipedia article for a list: https://en.wikipedia.org/wiki/CUDA
Build HipFT:
./build_nvidia_gpu.sh
Test the HipFT installation:
cd testsuite
./run_test_suite.sh
cd ..
You can use the "nvidia-smi" command in another terminal to make sure its computing on the GPU.
Return to the swqu_workshop/gpu
folder:
cd ../../
Install Solar Wind Generator (SWiG):
Create the SWiG installation folder:
mkdir swig
cd swig
Clone the git repository:
git clone --recursive https://github.com/predsci/swig.git/ .
Enter the MapFL directory and build MapFL:
cd mapfl
Copy the NVIDIA CPU build script:
cp build_examples/build_cpu_multithread_nvidia_ubuntu20.04.sh .
Open build_cpu_multithread_nvidia_ubuntu20.04.sh
and set the correct HDF5 lines as done above.
./build_cpu_multithread_nvidia_ubuntu20.04.sh
Test the MapFL installation:
cd example
../bin/mapfl mapfl_get_simple_open_field_map.in
If the run is sucessful, there should be a file called ofm.h5
in the directory.
Enter the POT3D directory and build POT3D:
cd ../pot3d
Copy the GPU build script:
cp build_examples/build_gpu_nvidia_ubuntu20.04.sh .
Open the build_gpu_nvidia_ubuntu20.04.sh
and set the correct HDF5 lines as done above.
Set the correct "cc" in the GPU lines (there are two). See the CUDA wikipedia article for a list: https://en.wikipedia.org/wiki/CUDA
Build POT3D:
./build_gpu_nvidia_ubuntu20.04.sh
Test the POT3D installation:
./validate.sh
The validation script will indicate if the run has passed.
NOTE! It is possible the second validation run will fail, even if the first one passes.
If this happens, than everything is good and POT3D should work fine for SWQU.
Add the following lines to the load_swqu_gpu
startup script:
export PATH="<DIR>/swqu_workshop/gpu/oft/bin:$PATH"
export PATH="<DIR>/swqu_workshop/gpu/oft/hipft/bin:$PATH"
export PATH="<DIR>/swqu_workshop/gpu/swig:$PATH"
export PATH="<DIR>/swqu_workshop/gpu/swig/bin:$PATH"
export PATH="<DIR>/swqu_workshop/gpu/swig/pot3d/bin:$PATH"
export PATH="<DIR>/swqu_workshop/gpu/swig/pot3d/scripts:$PATH"
where <DIR>
is the path to the directory that swqu_workshop
was made in.