Chapter 1 - Distributed Memory Programming with MPI

What is MPI?

Recall we have two main memory storing mechanics, distributed and shared. A distributed-memory system consists of a collection of core memory pairs connected by a network, and the memory associated with a core is directly accessible only to that core(figure 3.1), therefore, we need to use Message-Passing Interface to let the cores communicate

figure 3.1 distributed memory

MPI is not a language, is just a library that can be used in C/C++ and Fortran

---

Hello World!

Let's write a program similar to "hello world", instead, we can let each process simply print a message such that one process will share the output and another will print.

In a parallel world, processes can be identified as nonnegative integer ranks, so if there are p processes, then it will be 0,1,2,...,p-1 ranks.

#include <stdio.h>
#include <string.h>
#include <mpi.h>

const int MAX_STRING = 100;

int main(void){
    char greeting[MAX_STRING]; // Greeting string
    int comm_sz; // Number of processes
    int my_rank; // My process rank

    MPI_Init(NULL, NULL);
    MPI_Comm_size(MPI_COMM_WORLD, &comm_sz); // Initalize the number or processes
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); // Initalize the ranks

    if (my_rank != 0){ // master process is 0, this means that if the current process is not a master, then do this
        sprintf(greeting, "Greetings from process %d of %d!", my_rank, comm_sz); // Fill the greeting string
        MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD); // Send the greeting string from a process
    }
    else{
        printf("Greetings from process %d of %d!\n", my_rank, comm_sz); // Print the master process first
        for (int q = 1; q < comm_sz; q++){ // For every other processes that is not the master process
            MPI_Recv(greeting, MAX_STRING, MPI_CHAR, q, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE); // Master process receives the greeting string from a process and prints it    
            printf("%s\n", greeting);
        }
    }
    MPI_Finalize();
    return 0;
}

How to run it?

mpicc -g -Wall -o mpi_hello mpi_hello.c

-g produces debug info -Wall turns on all warnings -o mpi_hello creates an executable file and mpi_hello.c is the source file where it contains the code above

mpiexe -n 4 ./mpi_hello

-n 4 creates 4 processes, with the output similar to

Greetings from process 0 of 4!
Greetings from process 1 of 4!
Greetings from process 2 of 4!
Greetings from process 3 of 4!

MPI_Init

In Line 12, the call to MPI_Init tells the MPI system to do all of the necessary setup. For example, it might allocate storage for message buffers, and it might decide which process gets which rank. As a rule of thumb, no other MPI functions should be called before the program calls MPI_Init.

The fields are pointers to arguments, when we don't use them we can just set them as null

MPI_Finalize

In line 27, after we are finished with the MPI program we need to tell the MPI system to end this program, freeing any resources that are allocated for MPI, in general, no MPI functions should be called after MPI_Finalize

Communicators

In MPI a communicator is a collection of processes that can send messages to each other. One of the purposes of MPI_Init is to define a communicator that consists of all of the processes started by the user when she started the program. This communicator is called MPI_COMM_WORLD. The function calls in Lines 13 and 14 are getting information about MPI_COMM_WORLD.

SPMD Program

Notice that we compiled a single program—we didn’t compile a different program for each process—and we did this in spite of the fact that process 0 is doing something fundamentally different from the other processes: it’s receiving a series of messages and printing them, while each of the other processes is creating and sending a message. This is quite common in parallel programming. In fact, most MPI programs are written in this way. That is, a single program is written so that different processes carry out different actions, and this can be achieved by simply having the processes branch on the basis of their process rank. Recall that this approach to parallel programming is called single program, multiple data or SPMD. The if −else statement in Lines 16–29 makes our program SPMD.

MPI_Send

int MPI_Send(
    void* msg_buf_p,       /* in */,
    int msg_size,          /* in */,
    MPI_Datatype msg_type, /* in */,
    int dest,              /* in */,
    int tag,               /* in */,
    MPI_Comm communicator  /* in */
);

The first three arguments, msg_buf_p, msg_size, and msg_type, determine the contents of the message. The remaining arguments, dest, tag, and communicator, determine the destination of the message. The first argument, msg_buf_p, is a pointer to the block of memory containing the contents of the message.

• msg_buf_p: Pointer to the buffer that contains the message data.

• msg_size: Number of elements in the message.

• msg_type: MPI datatype of each message element.

• dest: Rank of the destination process within the communicator.

• tag: Message tag to help distinguish messages.

• communicator: MPI communicator used for the communication.

MPI_Recv

int MPI_Recv(
    void* msg_buf_p,       /* out */,
    int buf_size,          /* in */,
    MPI_Datatype buf_type, /* in */,
    int source,            /* in */,
    int tag,               /* in */,
    MPI_Comm communicator, /* in */,
    MPI_Status* status_p   /* out */
);

• msg_buf_p: Pointer to the buffer where the received message data will be stored.

• buf_size: Maximum number of elements in the buffer.

• buf_type: MPI datatype of each buffer element.

• source: Rank of the source process within the communicator.

• tag: Message tag to match the send operation.

• communicator: MPI communicator used for the communication.

• status_p: Pointer to an MPI_Status structure that will store information about the received message, such as the source, tag, and actual number of received elements.

This function will block until the data has been received from the source.