Parallel Computation, Message Pasing, OpenMP, dan CUDA GPU

Parallel computing is the use of more than one CPU to run a program simultaneously. Ideally, parallel processing makes the program run faster because more CPUs are used, unlike single computing that only use one CPU. But in practice to divide the program so that it can be executed by different CPUs without any interrelationship between them it’s usually difficult, so the program runs with multiple CPUs simultaneously in order to make the program better and can be processed quickly. It can be concluded that in parallel processing is different from the term multitasking, that means one CPU handles or executes several programs at once, parallel processing can be called also with the term parallel computing. Then for the difference between a single computing with parallel computing, it can be illustrated in the image below:

Image 1 Solving A Problem on Single Computing

Image 2 Solving A Problem on Parallel Computing

Types of Parallel Computers 

Based on hardware levels that support parallelism, in general the parallel computers can be classified: 

1. Multicore processing 

It is a processor that has several executing units. A multicore processor can perform several instructions per cycle from multiple streams of instructions. 

2. Symmetric multiprocessing 

A computer system with multiple identical processors that can use a shared memory or separate memory structure that is connected by bus. 

3. Distributed computing 

A computer system with distributed memory, where each processing element is connected by the network. 

4. Cluster computing 

Is a set of computers that work together and connected by the network, so it can be seen as a whole, this computer cluster is coordinated by a parent computer that tasked to distribute the work to each other computer. 

5. Massive parallel processing 

A single computer with multiple processors connected in a network. Inside the MPP each CPU has the same memory, operating system and application. Each subsystem communicating with each other through high-speed interconnects. 

6. Grid computing 

Grid computing utilizes the Internet in communicating between computers to solve a problem and is the most distributed parallel. 

7. Specialized parallel computer Parallel computer that serves to complete special tasks.

A. Parallelism Concept 

Parallel computing is one of computational techniques, where the computing process is done by several independent resources (computers) simultaneously. Parallel computing is usually required at the time of processing of large amounts of data or in the process of computing a very large number. In solving a problem, parallel computing requires a parallel machine infrastructure consisting of multiple computers connected to a network and capable of working in parallel, for that reason parallel computing need support software. 

Parallel programming is a computer programming technique that allows the execution of commands / operations simultaneously, either in computers with one or more CPUs. The main purpose of parallel programming is to improve computing performance. The more things you can do simultaneously (in the same time), the more work you can accomplish. 

An example of using this concept in everyday life is at the entrance / exit of the expressway (highway).


B. Distributed Processing 

Distributed processing is a process of distributing parallel processing in parallel processing using multiple machines. So, it can be say the ability of a computer that run simultaneously to solve a problem with a fast process. 

Distributed parallel processing using parallel processing on multiple machines. One example is how some communities allow users to register and dedicate their own computers to process some of the data sets given to them by the server. When thousands of users sign up, a lot of data can be processed in a very short amount of time.

C. Architectural Parallel Computer 

1. Single Instruction Single Data Stream (SISD) 

This model used 1 processor only. Therefore this model can be regarded as a model for single computing. In this type of computer all instructions are done sequentially one by one, but also possible overlapping (overlapping function used) in the execution of each section of instruction (pipelining). In general SISD computer in the form of a computer that consists of one processing (single processor). However, SISD computers may also have more than one functional unit (memory module, processing unit, etc.), as long as the entire functional unit in control of a control unit. An example of a SISD machine is a traditional PC or an old mainframe.

2. Single Instruction Multiple Data Stream (SIMD)

On a SIMD computer there is more than one processing element controlled by a single control unit. All processing elements receive and execute the same instructions that the controller sends, but to perform operations against different data coming from different data streams too. An example of a SIMD is an array processor (array processor) or GPU.

3. Multiple Instruction Single Data Stream (MISD) 

This type of computer has n processing unit that each receives and operates different instructions on the same data stream, because each processing unit has different controlling units. The output from one processor becomes input for the next processor. There is no real manifestation of this type of computer except in the form of prototype for research and never released to public.

4. Multiple Instruction Multiple Data Stream (MIMD) 

In this MIMD computer system there is an interaction between n processors. This is because all data streams from memory dank e, come from the same data space for all processors. The MIMD computer is tightly coupled (global memory) if the interaction between high processing and loosely coupled if the interaction rate between the processors is low. This computer has several autonomous processors capable of performing different instructions on different data. Distributed systems are generally known as MIMDs, either using a shared memory space or a distributed memory space.

D. Pengantar Thread Programming

A thread in computer programming is a related information about the use of a single program that can handle multiple users simultaneously. This Thread allows the program to know how the user entering the program turn in and the user will re-enter using a different user. Multiple threads can run simultaneously with other processes divide the resource into memory, while other processes do not divide it. Thread programming is divided into 2 :

1. Static Threading 

This technique is commonly used for computers with multiprocessors chips and other types of shared-memory computers. This technique allows the shared memory thread to be available, using the program counter and execute the program independently. The operating system places one thread on the processor and exchanges it with another thread that wants to use the processor. 

2. Dynamic Multithreading 

This technique is a development of previous techniques that aim for convenience because with it, programmers do not have to bother with communication protocols, load balancing, and other complexity that exist in static threading. This concurrency platform provides scheduler that load balacing automatically. Although the platform is still under development but generally supports two features: nested parallelism and parallel loops.

E. Introduction Message Pasing, Open MP 

Message Passing is a form of communication used in parallel computing, OOT (Object Oriented Programming) or Object-Based Programming and interprocess communication. MPI is a programming standard that allows programmers to create an application that can run in parallel. Processes run by an application can be shared to send for each compute node which then each compute node is processed and returns the result to the computer head node.

Other uses of MPI are : 

1. Writes parallel code in portable form 

2. Get high performance in parallel programming 

3. Face problems that involving irregular or dynamic data relationships that are’nt very compatible with parallel data models.

OpenMP Consists of a set of compiler commands, library routines, and environment variables that affect run-time. Many applications built with parallel programming hybrid models can run on cluster computers using OpenMP and Message Passing Interface (MPI), or more transparently using OpenMP extensions of non-shared memory systems.

F. Pengantar Pemograman CUDA GPU 

Before entering CUDA discussion we have to learn GPU first. GPU is a special processor to speed up and change memory to speed up image processing. GPU itself is usually located in the computer graphics card or laptop.

CUDA (Compute Unified Device Architecture) is a scheme created by NVIDIA for NVIDIA as a GPU (Graphic Processing Unit) that capable of computing not only for graphics processing but also for general purpose. So with the CUDA we can take advantage of many processors from NVIDIA to perform the calculation process or even computing a lot.