This section describes some debug situations within the context of a single process composed of many OpenMP threads. An attempt is made by PGDBG to preserve line level debugging and to help make debugging OpenMP programs more intuitive. This section describes how PGDBG controls OpenMP threads in the following situations:
As described below, threads may be held and others advanced automatically to negotiate OpenMP program constructs like synchronization points and critical sections. PGDBG also sets the thread stop mode to synchronous when a program runs to a serial region, and asynchronous when a program runs to a parallel region. These features are an attempt to make the process of debugging an OpenMP program easier, and are included by default. (See Disabling PGDBG's OpenMP Event Support to turn off this behavior). PGDBG assumes a legal OpenMP target program.
A control command applied to a running process will only be applied to the stopped threads of that process, and is ignored by its running threads. Threads held by the PGDBG OpenMP event handler will also ignored the control command in most situations.
The initial thread is the thread with OpenMP ID 0.
Conceptually, the initial thread is the only thread that is well defined (for the purpose of doing useful work) in a serial region of code. All threads are well defined in a parallel region of code. When the initial thread is in a serial region, the non-initial threads are busy waiting at the end of the last parallel region, waiting to be called down to do some work in the next parallel region. All threads enter the (next) parallel region only when the first thread has entered the parallel region, i.e. the initial thread is not in a serial region.
PGDBG source line level debugging operations (next, step,...) are not well defined for non-initial threads in serial regions since these threads are stuck in a busy loop, which is not compiled to include source line information. The instruction level PGDBG control commands (nexti, stepi, ...) are well defined if you want to advance through the described wait loop at the assembly level.
Example:
if next is applied to a single thread with OpenMP ID 3 (non-initial thread) in a serial region and all other threads are stopped, then next will never complete. By definition next returns only when the thread hits a source line, but thread 3 running by itself in a serial region will never hit a text address that corresponds to a source line since it is waiting in a busy loop for work to do in the next parallel region. The program will not run to the next parallel region unless thread 0 is run. The busy loop is not compiled with debug information, and is linked into your program when you compile with -mp or -Mconcur.
Control commands should only be applied to the initial thread in serial regions of code. Synchronous thread stop mode can also be used, however 'stepping' and 'nexting' all threads in a serial region may slow down the debugger.
We now describe some common debug situations in the context of OpenMP debugging.
In the following example the initial thread is stopped in a serial region before entering its first parallel region. To start, the debugger is operating in serial mode. The prompt does not display any information about threads or p/t-sets accordingly (See The PGDBG Command Prompt)
Example:
Breakpoint at 0x8048fe0, function main, file omp.c, line 7
#7: #pragma omp parallel
pgdbg> list
#2:
#3: main(){
#4:
#5: printf("One thread ... \n");
#6:
#7:==>> #pragma omp parallel
#8: {
#9: int myid,i;
#10:
#11: myid = omp_get_thread_num();
pgdbg> n
(Entering Parallel Region)
Warning: OMP_NUM_THREADS greater than available cpus (set to 4; cpus = 1)
([1] New Thread)
([2] New Thread)
([3] New Thread)
[0] Stopped at 0x8048ffe, function main, file omp.c, line 11
#11: myid = omp_get_thread_num();
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10844 Stopped SIGTRAP main line: 11 in "omp.c" address: 0x8048ffe
1 10846 Stopped SIGTRAP main line: 11 in "omp.c" address: 0x8048ffe
2 10847 Stopped SIGTRAP main line: 11 in "omp.c" address: 0x8048ffe
3 10848 Stopped SIGTRAP main line: 11 in "omp.c" address: 0x8048ffe
pgdbg [all] 0>
We 'next' the current thread into the parallel region. The debugger enters threads-only mode automatically. The current p/t-set is described by the prompt to be [all], which is the default debugger-defined, p/t-set for "all threads, all processes". The prompt also describes which thread is the current thread, thread 0.
See Status Messages for how to enable OpenMP event messages. PGDBG prints a message describing that we are entering a parallel region. This is useful when debugging auto-parallelized programs (compiled with -Mconcur).
If the threads have not been created yet, they are created and PGDBG automatically attaches to them, printing a status message for each thread.
All threads inherit the control command of the current thread (in this case thread 0) upon entering a parallel region. Unless overridden by the user, PGDBG sets the thread stop mode to asynchronous upon entering the parallel region. As a result all threads are stopped at line 25 inside the parallel region, the first source line in the execution path of each.
Continuing from above with all threads stopped in the parallel region:
pgdbg [all] 0.0> list
#6:
#7: #pragma omp parallel
#8: {
#9: int myid,i;
#10:
#11:==>> myid = omp_get_thread_num();
#12:
#13: for(i=0;i<2;i++){
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
#15: }
pgdbg [all] 0> n
[3] Stopped at 0x8049007, function main, file omp.c, line 13
#13: for(i=0;i<2;i++){
[2] Stopped at 0x8049007, function main, file omp.c, line 13
#13: for(i=0;i<2;i++){
[1] Stopped at 0x8049007, function main, file omp.c, line 13
#13: for(i=0;i<2;i++){
[0] Stopped at 0x8049007, function main, file omp.c, line 13
#13: for(i=0;i<2;i++){
([*] Process Stopped)
pgdbg [all] 0> [all] n
[3] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
[2] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
[1] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
[0] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
([*] Process Stopped)
pgdbg [all] 0> [*] n
HELLO 10857, 3
HELLO 10856, 2
HELLO 10855, 1
HELLO 10853, 0
[3] Stopped at 0x804903a, function main, file omp.c, line 15
#15: }
[2] Stopped at 0x804903a, function main, file omp.c, line 15
#15: }
[1] Stopped at 0x804903a, function main, file omp.c, line 15
#15: }
[0] Stopped at 0x804903a, function main, file omp.c, line 15
#15: }
([*] Process Stopped)
pgdbg [all] 0> [2:3] n
[3] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
[2] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
([*] Process Stopped)
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10853 Stopped SIGTRAP main line: 15 in "omp.c" address: 0x804903a
1 10855 Stopped SIGTRAP main line: 15 in "omp.c" address: 0x804903a
2 10856 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
3 10857 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
pgdbg [all] 0> focus [0:1]
[0:1] : [0,1]
pgdbg [0:1] 0> n
[1] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
[0] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
([*] Process Stopped)
pgdbg [0:1] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10853 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
1 10855 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
2 10856 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
3 10857 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
pgdbg [0:1] 0> thread 1
[1] Stopped at 0x8049016, function main, file omp.c, line 14
#14: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
([1] Thread Stopped)
pgdbg [0:1] 1> list
#9: int myid,i;
#10:
#11: myid = omp_get_thread_num();
#12:
#13: for(i=0;i<2;i++){
#14:==>> printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
#15: }
#16:
#17: }
#18:
pgdbg [0:1] 1> threads
0 ID PID STATE SIGNAL LOCATION
0 10853 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
=> 1 10855 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
2 10856 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
3 10857 Stopped SIGTRAP main line: 14 in "omp.c" address: 0x8049016
pgdbg [0:1] 1> print myid
1
pgdbg [0:1] 1> [*] print myid
[0] print myid:
0
[1] print myid:
1
[2] print myid:
2
[3] print myid:
3
The prompt indicates that the current p/t-set is [all], the debugger-defined set for [*]. From line 11 we 'next' all threads to line 13. Notice that every thread has been advanced to the next source line. This is because the target p/t-set is [all], the current p/t-set, by default. In synchronous thread stop mode at least one thread would have advanced to the next line. In asynchronous stop mode all threads are advanced to the next line. PGDBG enters asynchronous stop mode automatically (see the pgienv command).
The following commands are equivalent in this example:
pgdbg [all] 0> n pgdbg [all] 0> [all] n pgdbg [all] 0> [*] n
We then use the prefix p/t-set [2:3] to advance threads 2 and 3 only. Using a prefix p/t-set overrides the default current p/t-set.
The focus command can be used to set the current p/t-set. In the absence of a prefix p/t-set, the current p/t-set is used as the target or default p/t-set.
Use the thread command to change the current thread. The debugger enters into the scope of the new current thread.
Continuing from above.
pgdbg [0:1] 1> n
[0] Stopped at 0x8049045, function main, file omp.c, line 17
#17: }
[1] Stopped at 0x8049045, function main, file omp.c, line 17
#17: }
([*] Process Stopped)
pgdbg [0:1] 1> n
HELLO 10857, 3
HELLO 10856, 2
[3] Stopped at 0x8049ec0, function _mp_pexit
8049ec0: 58 popl %eax
[2] Stopped at 0x8049ec0, function _mp_pexit
8049ec0: 58 popl %eax
([*] Entering Serial Region)
[0] Stopped at 0x804904f, function main, file omp.c, line 19
#19: printf("... back to one thread.\n");
[1] Stopped at 0x8049f9c, function _mp_slave
8049f9c: eb 82 jmp 0x82 <8049f20>
([*] Process Stopped)
pgdbg [0:1] 1> thread 0
[0] Stopped at 0x804904f, function main, file omp.c, line 19
#19: printf("... back to one thread.\n");
([0] Thread Stopped)
pgdbg [0:1] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10853 Stopped SIGTRAP main line: 19 in "omp.c" address: 0x804904f
1 10855 Stopped SIGTRAP _mp_slave address: 0x8049f9c
2 10856 Stopped SIGTRAP _mp_slave address: 0x8049f9c
3 10857 Stopped SIGTRAP _mp_slave address: 0x8049f9c
pgdbg [0:1] 0> focus [0]
[0] : [0]
[0]
pgdbg [0] 0> n
... back to one thread.
[0] Stopped at 0x804905f, function main, file omp.c, line 21
#21: return;
([*] Process Stopped)
pgdbg [0] 0> ...
We `next' out of the parallel region stopping in a serial region. The non-initial threads (threads 1,2,3 are stopped in their wait loop), and the initial thread has entered a serial region. In order for the initial thread to enter a serial region, all non-initial threads must finish their work in the current parallel region. PGDBG coordinates threads in this way. When a thread hits the end of a parallel region, and the current control command would require a thread to leave the parallel region, PGDBG synchronizes all threads to the end of the parallel region, sets the thread stop mode to synchronous (unless overridden previously by the user using the pgienv command), and copies the control command to all threads. Only the initial thread will enter the serial region (by OpenMP semantics) and stop once the control command is satisfied. For non-initial threads, line 19 is not well defined for the purpose of doing useful work.
Upon entering a serial region we change the current thread to be the initial thread (0) and set the current p/t-set to focus on thread 0 by default. Advancing all threads together in a serial region may slow down the debugger.
Example:
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10897 Stopped SIGTRAP ff line: 38 in "ompsync.c" address: 0x804904b
1 10899 Stopped SIGTRAP ff line: 38 in "ompsync.c" address: 0x804904b
2 10900 Stopped SIGTRAP ff line: 38 in "ompsync.c" address: 0x804904b
3 10901 Stopped SIGTRAP ff line: 38 in "ompsync.c" address: 0x804904b
pgdbg [all] 0> list
#33: /*test sync in par region*/
#34: #pragma omp parallel
#35: {
#36: int i,sum;
#37:
#38:==>> printf("HELLO from %d\n",omp_get_thread_num());
#39: fflush(stdout);
#40:
#41: #pragma synchronize
#42: for (i=0;i<20000;i++)
pgdbg [all] 0> focus [2]
WARNING: Current thread 0 is not in target p/t-set.
[2] : [2]
[2]
pgdbg [2] 0> s
HELLO from 2
[2] Stopped at 0x804905e, function ff, file ompsync.c, line 39
#39: fflush(stdout);
([*] Process Stopped)
pgdbg [2] 0> s
[2] Stopped at 0x804906f, function ff, file ompsync.c, line 42
#42: for (i=0;i<20000;i++)
([*] Process Stopped)
pgdbg [2] 0> s
HELLO from 0
HELLO from 1
HELLO from 3
[3] Stopped at 0x804945a, function _mp_barrier2
804945a: 83 3d 74 c7 04 08 02 cmpl $0x2,0x804c774
[1] Stopped at 0x804945a, function _mp_barrier2
804945a: 83 3d 74 c7 04 08 02 cmpl $0x2,0x804c774
([*] Synchronizing)
[0] Stopped at 0x8049085, function ff, file ompsync.c, line 43
#43: sum+=i;
[3] Stopped at 0x8049085, function ff, file ompsync.c, line 43
#43: sum+=i;
[2] Stopped at 0x8049085, function ff, file ompsync.c, line 43
#43: sum+=i;
[1] Stopped at 0x8049085, function ff, file ompsync.c, line 43
#43: sum+=i;
([*] Process Stopped)
pgdbg [2] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10905 Stopped SIGTRAP ff line: 43 in "ompsync.c" address: 0x8049085
1 10907 Stopped SIGTRAP ff line: 43 in "ompsync.c" address: 0x8049085
2 10908 Stopped SIGTRAP ff line: 43 in "ompsync.c" address: 0x8049085
3 10909 Stopped SIGTRAP ff line: 43 in "ompsync.c" address: 0x8049085
pgdbg [2] 0> list
#38: printf("HELLO from %d\n",omp_get_thread_num());
#39: fflush(stdout);
#40:
#41: #pragma synchronize
#42: for (i=0;i<20000;i++)
#43:==>> sum+=i;
#44:
#45: #pragma synchronize
#46: printf("HELLO from %d\n",omp_get_thread_num());
#47: fflush(stdout);
pgdbg [2] 0>
With verbose messaging turned on we start with all threads stopped in a parallel region. We focus on thread 2 by making it the only member in the current p/t-set. We advance thread 2 over an OpenMP synchronize statement. PGDBG synchronizes all threads over the OpenMP synchronize even though the 'next' command was applied only to thread 2. This is because in order for thread 2 to reach line 42 it must cross an OpenMP synchronize statement, which by definition requires all threads to have finished their work before the synchronize in order for any thread to cross over it.
The sync, and synci commands provide a way to synchronize a group of threads to a particular program location.
A critical section is a section of code into which at most one thread may enter at a time. Critical sections are embedded in parallel sections of code. A serial section could be looked at as a critical section into which only the initial thread may enter. A thread waiting to enter the critical section waits in a busy loop. A thread continues on its way upon exiting the critical section.
Because of the busy waiting involved, it is best to debug critical sections with PGDBG's thread stop mode set to synchronous (threads stop together). Use the pgienv command to configure the stop mode. This way when the next thread stops in the critical section, all other threads will be directed to stop. Alternately you could also use 'pgienv threadwait none' and enter control commands while some threads are running (halt, or continue etc.)
Example: Using synchronous thread stop mode
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10922 Stopped SIGTRAP main line: 37 in "crit.c" address: 0x8049078
1 10924 Stopped SIGTRAP main line: 37 in "crit.c" address: 0x8049078
2 10925 Stopped SIGTRAP main line: 37 in "crit.c" address: 0x8049078
3 10926 Stopped SIGTRAP main line: 37 in "crit.c" address: 0x8049078
pgdbg [all] 0> list
#32: /*test sync in par region*/
#33: #pragma omp parallel
#34: {
#35: int i,sum;
#36:
#37:==>> printf("HELLO from %d\n",omp_get_thread_num());
#38: fflush(stdout);
#39:
#40: #pragma critical
#41: {
pgdbg [all] 0> list
#42: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#43: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#44: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#45: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#46: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#47: }
#48:
#49: printf("HELLO from %d\n",omp_get_thread_num());
#50: fflush(stdout);
#51:
pgdbg [all] 0> pgienv threadstopconfig user
Entering user defined thread stop mode.
pgdbg [all] 0> pgienv threadstop sync
Threads will stop synchronously.
pgdbg [all] 0> b 45
[all] breakpoint set at: main line: 45 in "crit.c" address: 0x80490e5
2
pgdbg [all] 0> c
HELLO from 0
INSIDE CRITICAL 0
INSIDE CRITICAL 0
INSIDE CRITICAL 0
HELLO from 3
HELLO from 1
HELLO from 2
[0] Breakpoint at 0x80490e5, function main, file crit.c, line 45
#45: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
([*] Process Stopped)
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10969 Stopped SIGTRAP main line: 45 in "crit.c" address: 0x80490e5
1 10971 Stopped SIGSTOP _mp_p address: 0x804a189
2 10972 Stopped SIGSTOP _mp_p address: 0x804a189
3 10973 Stopped SIGSTOP _mp_p address: 0x804a189
To enter a critical section, set a breakpoint inside it. Continue all threads "into" the section. At most one will enter. If PGDBG is in asynchronous thread stop mode (the default in parallel sections) you will have to explicitly stop the other threads using the halt command. In order to enter commands while threads are running set the pgienv wait modes to none. The rest of the threads will be busy waiting at the entrance to the critical section. If PGDBG is in synchronous thread stop mode all threads will stop when one thread stops.
Thread 0 has entered the critical region and hit breakpoint at line 45. The other threads are stopped in a busy loop waiting for thread 0 to leave the critical section.
Set a breakpoint at line 50 to catch threads as they leave the critical section and continue all threads.
pgdbg [all] 0> b 50
[all] breakpoint set at: main line: 50 in "crit.c" address: 0x804912e
3
pgdbg [all] 0> c
INSIDE CRITICAL 0
INSIDE CRITICAL 0
HELLO from 0
INSIDE CRITICAL 3
INSIDE CRITICAL 3
INSIDE CRITICAL 3
[3] Breakpoint at 0x80490e5, function main, file crit.c, line 45
#45: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
[0] Breakpoint at 0x804912e, function main, file crit.c, line 50
#50: fflush(stdout);
([*] Process Stopped)
pgdbg [all] 0>
Thread 0 has exited the critical region and thread 3 has entered the critical region. Threads 1 and 2 are busy waiting for entry.
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 10969 Stopped SIGTRAP main line: 50 in "crit.c" address: 0x804912e
1 10971 Stopped SIGSTOP _mp_p address: 0x804a180
2 10972 Stopped SIGSTOP _mp_p address: 0x804a189
3 10973 Stopped SIGTRAP main line: 45 in "crit.c" address: 0x80490e5
To iterate all threads into the critical section, continue all threads that have not stopped at the breakpoint set after the critical section. This will continue the thread currently stopped in the critical section and those that have not yet entered.
When the desired thread has entered the critical region apply control commands to that thread only.
Example: Using thread stop mode async, thread wait mode none
A control command can be applied to the stopped threads of a running process. The control command is ignored by any running threads. In the following example we set the thread wait mode to none and set the thread stop mode to async. We are then able to iterate through the threads asynchronously.
pgdbg> pgienv threadstopconfig user
Entering user defined thread stop mode.
pgdbg> pgienv threadstop async
Threads will stop asynchronously.
pgdbg> pgienv threadwait none
Wait for NO threads to stop before prompt is available.
pgdbg> list 40,54
#40: #pragma critical
#41: {
#42: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#43: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#44: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#45: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#46: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#47: }
#48:
#49: printf("HELLO from %d\n",omp_get_thread_num());
#50: fflush(stdout);
#51:
#52: #pragma synchronize
#53: printf("HELLO from %d\n",omp_get_thread_num());
#54: fflush(stdout);
pgdbg> b 45
breakpoint set at: main line: 45 in "crit.c" address: 0x80490e5
1
pgdbg> b 54
breakpoint set at: main line: 54 in "crit.c" address: 0x8049157
2
pgdbg [all] 0> run
reloading ./crit
argv[0]= ./crit
./crit loaded by ld-linux.so.2.
libpgthread.so loaded by ld-linux.so.2.
libm.so.6 loaded by ld-linux.so.2.
libc.so.6 loaded by ld-linux.so.2.
ld-linux.so.2 loaded by ld-linux.so.2.
One thread ...
([*] Entering Parallel Region)
Warning: OMP_NUM_THREADS greater than available cpus (set to 4; cpus = 1)
([1] New Thread)
([2] New Thread)
([3] New Thread)
HELLO from 0
INSIDE CRITICAL 0
INSIDE CRITICAL 0
INSIDE CRITICAL 0
[0] Breakpoint at 0x80490e5, function main, file crit.c, line 45
#45: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
HELLO from 1
HELLO from 2
HELLO from 3
pgdbg [all] 0>c
[1] ERROR: Thread Running. Ignoring cont.
[2] ERROR: Thread Running. Ignoring cont.
[3] ERROR: Thread Running. Ignoring cont.
INSIDE CRITICAL 0
INSIDE CRITICAL 0
HELLO from 0
INSIDE CRITICAL 1
INSIDE CRITICAL 1
INSIDE CRITICAL 1
[1] Breakpoint at 0x80490e5, function main, file crit.c, line 45
#45: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
pgdbg [all] 0>c
[0] ERROR: Thread Stopped. Ignoring cont.
[2] ERROR: Thread Running. Ignoring cont.
[3] ERROR: Thread Running. Ignoring cont.
INSIDE CRITICAL 1
INSIDE CRITICAL 1
HELLO from 1
INSIDE CRITICAL 3
INSIDE CRITICAL 3
INSIDE CRITICAL 3
[3] Breakpoint at 0x80490e5, function main, file crit.c, line 45
#45: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
pgdbg [all] 0>c
[0] ERROR: Thread Stopped. Ignoring cont.
[1] ERROR: Thread Stopped. Ignoring cont.
[2] ERROR: Thread Running. Ignoring cont.
INSIDE CRITICAL 3
INSIDE CRITICAL 3
HELLO from 3
INSIDE CRITICAL 2
INSIDE CRITICAL 2
INSIDE CRITICAL 2
[2] Breakpoint at 0x80490e5, function main, file crit.c, line 45
#45: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
[0] Stopped at 0x80494da, function _mp_barrier2
80494da: 83 3d d4 c7 04 08 02 cmpl $0x2,0x804c7d4
([*] Process Stopped)
pgdbg [all] 0> c
INSIDE CRITICAL 2
INSIDE CRITICAL 2
HELLO from 2
([*] Synchronizing)
HELLO from 0
[0] Breakpoint at 0x8049157, function main, file crit.c, line 54
#54: fflush(stdout);
HELLO from 1
[1] Breakpoint at 0x8049157, function main, file crit.c, line 54
#54: fflush(stdout);
HELLO from 2
[2] Breakpoint at 0x8049157, function main, file crit.c, line 54
#54: fflush(stdout);
HELLO from 3
[3] Breakpoint at 0x8049157, function main, file crit.c, line 54
#54: fflush(stdout);
([*] Process Stopped)
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 11323 Stopped SIGTRAP main line: 54 in "crit.c" address: 0x8049157
1 11325 Stopped SIGTRAP main line: 54 in "crit.c" address: 0x8049157
2 11326 Stopped SIGTRAP main line: 54 in "crit.c" address: 0x8049157
3 11327 Stopped SIGTRAP main line: 54 in "crit.c" address: 0x8049157
pgdbg [all] 0>
PGDBG recognizes the synchronization pragma at line 52. When a thread reaches line 52, PGDBG holds it there in a stopped state until the rest of the threads 'sync' to that point. This behavior can be disabled. Set the omp environment variable to off using the pgienv command.
The omp sections pragma defines a non-iterative work-sharing construct within a parallel region. Each section is executed by a single thread. If there are more threads than sections, some threads will have no work and will jump to the implied barrier at the end of the construct. If there are more sections than threads, one or more threads will execute more than one section.
Example:
pgdbg> pgienv verbose -1
pgdbg> b 10
breakpoint set at: main line: 10 in "psec.c" address: 0x8048fd6
1
pgdbg> b 12
breakpoint set at: main line: 12 in "psec.c" address: 0x8048ff9
2
pgdbg> c
(Entering Parallel Region)
Warning: OMP_NUM_THREADS greater than available cpus (set to 4; cpus = 1)
([1] New Thread)
([2] New Thread)
([3] New Thread)
[1] Breakpoint at 0x8048ff9, function main, file psec.c, line 12
#12: printf("%d\n",omp_get_thread_num());
[0] Breakpoint at 0x8048fd6, function main, file psec.c, line 10
#10: printf("%d\n",omp_get_thread_num());
([*] Process Stopped)
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 11453 Stopped SIGTRAP main line: 10 in "psec.c" address: 0x8048fd6
1 11455 Stopped SIGTRAP main line: 12 in "psec.c" address: 0x8048ff9
2 11456 Stopped SIGTRAP _mp_pexit address: 0x8049e80
3 11457 Stopped SIGTRAP _mp_pexit address: 0x8049e80
pgdbg [all] 0> list
#5: printf("One thread ... \n");
#6:
#7: #pragma omp parallel sections
#8: {
#9: #pragma omp section
#10:==>> printf("%d\n",omp_get_thread_num());
#11: #pragma omp section
#12: printf("%d\n",omp_get_thread_num());
#13: }
#14:
Set a breakpoint across all threads in a parallel section and configure the thread wait and stop modes using pgienv. Set a breakpoint in each parallel section to debug parallel sections, or set the thread stop mode to sync using the pgienv command (you must also set threadstopconfig to user).
In the above example, threads 2 and 3 are held by PGDBG at _mp_pexit, the implicit barrier at the end of the parallel region. Thread 0 and thread 1 are each stopped in a parallel section.
The following pragmas define regions of code that execute exclusively on one thread.
#pragma omp single #pragma omp master
A breakpoint set in these regions will trigger for at most one thread.
Example:
pgdbg [all] 0> list
#6:
#7: #pragma omp parallel
#8: {
#9: int myid,i;
#10:
#11:==>> myid = omp_get_thread_num();
#12:
#13: #pragma omp master
#14: printf("This statement only executed by master (%d)\n",myid);
#15:
pgdbg [all] 0> list
#16: #pragma omp single nowait
#17: printf("This statement only executed by %d\n",myid);
#18:
#19: for(i=0;i<2;i++){
#20: printf("HELLO %d, %d\n",getpid(),omp_get_thread_num());
#21: }
#22:
#23: }
#24:
#25: printf("... back to one thread.\n");
pgdbg [all] 0> b 14
[all] breakpoint set at: main line: 14 in "single2.c" address: 0x804900f
2
pgdbg [all] 0> b 17
[all] breakpoint set at: main line: 17 in "single2.c" address: 0x8049031
3
pgdbg [all] 0> b 19
[all] breakpoint set at: main line: 19 in "single2.c" address: 0x8049043
4
We start this example with all threads stopped in the parallel region at line 11. We set 3 breakpoints. The first breakpoint at line 14 is only defined for thread 0 since line 14 is wrapped in a omp master pragma. This pragma states that line 14 will only be executed by thread 0. The second breakpoint is defined at line 17. It is defined for at most one thread since line 17 is wrapped in a omp single pragma. This pragma states that at most one thread will execute line 17. The third breakpoint is set at line 19 and is defined for all threads.
After setting the breakpoints we continue all threads.
pgdbg [all] 0> c
[0] Breakpoint at 0x804900f, function main, file single2.c, line 14
#14: printf("This statement only executed by master (%d)\n",myid);
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 11529 Stopped SIGTRAP main line: 14 in "single2.c" address: 0x804900f
1 11531 Stopped SIGTRAP main line: 17 in "single2.c" address: 0x8049031
2 11532 Stopped SIGTRAP main line: 19 in "single2.c" address: 0x8049043
3 11533 Stopped SIGTRAP main line: 19 in "single2.c" address: 0x8049043
The master thread, thread 0, is stopped at line 14. Thread 1 stops at line 17 by itself, and the rest of the threads stop at line 19.
Without the nowait clause, the omp single pragma contains an implicit barrier. PGDBG contains support to synchronize the threads over the barrier. The same example less the nowait clause yields the following behavior:
pgdbg [all] 0> c
[0] Breakpoint at 0x804900f, function main, file single.c, line 14
#14: printf("This statement only executed by master (%d)\n",myid);
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 11543 Stopped SIGTRAP main line: 14 in "single.c" address: 0x804900f
1 11545 Stopped SIGTRAP main line: 17 in "single.c" address: 0x8049031
2 11546 Stopped SIGTRAP _mp_barrier2 address: 0x80493aa
3 11547 Stopped SIGTRAP _mp_barrier2 address: 0x80493aa
Threads 2 and 3 are held at the barrier by PGDBG.
pgdbg [1] 0> focus [1]
[1] : [1]
[1]
pgdbg [1] 0> s
This statement only executed by 1
This statement only executed by master (0)
[0] Breakpoint at 0x8049048, function main, file single.c, line 19
#19: for(i=0;i<2;i++){
pgdbg [1] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 11543 Stopped SIGTRAP main line: 19 in "single.c" address: 0x8049048
1 11545 Stopped SIGTRAP main line: 19 in "single.c" address: 0x8049048
2 11546 Stopped SIGTRAP main line: 19 in "single.c" address: 0x8049048
3 11547 Stopped SIGTRAP main line: 19 in "single.c" address: 0x8049048
In the absence of events (breakpoints, signals) PGDBG synchronizes all threads over the barrier when thread 1 is continued. This behavior can be disabled using the pgienv command.
PGDBG provides explicit support for OpenMP events. OpenMP events are points in a well-defined OpenMP program where one thread depends on the program location of another thread for it to continue (a barrier for example). PGDBG's support for OpenMP events can be disabled using the omp pgienv environment variable.
pgienv omp [on|off]
If PGDBG's OpenMP Event support is disabled, (`pgienv omp off') it is recommended that you use the following pgienv settings:
pgienv threadstopconfig user pgienv threadstop async pgienv threadwait none
These settings will allow threads to stop independently, while allowing commands to be entered while threads are running (`threadwait none'). This will be necessary to start (cont, next, ...) and stop (halt) threads while some are spinning in OpenMP wait loops (barriers etc.).
Revisit the example from Critical Sections with OpenMP event support disabled.
pgdbg> pgienv omp off
OpenMP events disabled.
pgdbg> pgienv threadstopconfig user
Entering user defined thread stop mode.
pgdbg> pgienv threadstop async
Threads will stop asynchronously.
pgdbg> pgienv threadwait none
Wait for NO threads to stop before prompt is available.
pgdbg> list
#1: #include <stdio.h>
#2: #include <time.h>
#3:
#4: extern int omp_get_thread_num(), omp_get_num_threads();
#5:
#6: main(){
#7:
#8: #pragma omp parallel
#9: {
#10: #pragma critical
pgdbg> list
#11: {
#12: printf("INSIDE CRITICAL %d\n",omp_get_thread_num());
#13: fflush(stdout);
#14: }
#15: #pragma synchronize
#16: printf("HELLO from %d\n",omp_get_thread_num());
#17: fflush(stdout);
#18: }
#19: return;
#20: }
pgdbg> b 13
breakpoint set at: main line: 13 in "crit2.c" address: 0x8048fe1
1
pgdbg> b 17
breakpoint set at: main line: 17 in "crit2.c" address: 0x804901a
2
pgdbg> pgienv verbose -1
pgdbg> c
./crit2 loaded by ld-linux.so.2.
libpgthread.so loaded by ld-linux.so.2.
libm.so.6 loaded by ld-linux.so.2.
libc.so.6 loaded by ld-linux.so.2.
ld-linux.so.2 loaded by ld-linux.so.2.
Warning: OMP_NUM_THREADS greater than available cpus (set to 4; cpus = 1)
([1] New Thread)
([2] New Thread)
([3] New Thread)
INSIDE CRITICAL 0
[0] Breakpoint at 0x8048fe1, function main, file crit2.c, line 13
#13: fflush(stdout);
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 11671 Stopped SIGTRAP main line: 13 in "crit2.c" address: 0x8048fe1
1 11673 Running
2 11674 Running
3 11675 Running
pgdbg [all] 0> c
[1] ERROR: Thread Running. Ignoring cont.
[2] ERROR: Thread Running. Ignoring cont.
[3] ERROR: Thread Running. Ignoring cont.
INSIDE CRITICAL 2
[2] Breakpoint at 0x8048fe1, function main, file crit2.c, line 13
#13: fflush(stdout);
pgdbg [all] 0> c
[0] ERROR: Thread Running. Ignoring cont.
[1] ERROR: Thread Running. Ignoring cont.
[3] ERROR: Thread Running. Ignoring cont.
INSIDE CRITICAL 3
[3] Breakpoint at 0x8048fe1, function main, file crit2.c, line 13
#13: fflush(stdout);
pgdbg [all] 0> c
[0] ERROR: Thread Running. Ignoring cont.
[1] ERROR: Thread Running. Ignoring cont.
[2] ERROR: Thread Running. Ignoring cont.
INSIDE CRITICAL 1
[1] Breakpoint at 0x8048fe1, function main, file crit2.c, line 13
#13: fflush(stdout);
pgdbg [all] 0> threads
0 ID PID STATE SIGNAL LOCATION
=> 0 11671 Running
1 11673 Stopped SIGTRAP main line: 13 in "crit2.c" address: 0x8048fe1
2 11674 Running
3 11675 Running
pgdbg [all] 0> c
[0] ERROR: Thread Running. Ignoring cont.
[2] ERROR: Thread Running. Ignoring cont.
[3] ERROR: Thread Running. Ignoring cont.
HELLO from 0
[0] Breakpoint at 0x804901a, function main, file crit2.c, line 17
#17: fflush(stdout);
HELLO from 1
[1] Breakpoint at 0x804901a, function main, file crit2.c, line 17
#17: fflush(stdout);
HELLO from 2
[2] Breakpoint at 0x804901a, function main, file crit2.c, line 17
#17: fflush(stdout);
HELLO from 3
[3] Breakpoint at 0x804901a, function main, file crit2.c, line 17
#17: fflush(stdout);
([*] Process Stopped)
A breakpoint is set in a critical section at line 13. Each thread stops at line 13 as it enters the critical section, the other threads are left running in their busy wait loops (either before the critical section, or before at the synchronization pragma at line 15). A thread stopped at the breakpoint at line 13 prevents all threads from hitting the breakpoint at line 17 by the semantics of the synchronize pragma.