linux - High Kernel CPU when running multiple python progams -


i developed python program heavy numerical calculations. run on linux machine 32 xeon cpus, 64gb ram, , ubuntu 14.04 64-bit. launch multiple python instances different model parameters in parallel use multiple processes without having worry global interpreter lock (gil). when monitor cpu utilization using htop, see cores used, of time kernel. generally, kernel time more twice user time. i'm afraid there lot of overhead going on on system level, i'm not able find cause this.

how 1 reduce high kernel cpu usage?

here observation made:

  • this effect appears independent of whether run 10 jobs or 50. if there fewer jobs cores, not cores used, ones used still have high cpu usage kernel
  • i implemented inner loop using numba, problem not related this, since removing numba part not resolve problem
  • i though might related using python2 similar problem mentioned in question switching python2 python3 did not change much
  • i measured total number of context switches performed os, 10000 per second. i'm not sure whether large number
  • i tried increasing python time slices setting sys.setcheckinterval(10000) (for python2) , sys.setswitchinterval(10) (for python3) none of helped
  • i tried influencing task scheduler running schedtool -b pid didn't help

edit: here screenshot of htop: enter image description here

i ran perf record -a -g , report perf report -g graph:

samples: 1m of event 'cycles', event count (approx.): 1114297095227                                    -  95.25%          python3  [kernel.kallsyms]                           [k] _raw_spin_lock_irqsave   ◆    - _raw_spin_lock_irqsave                                                                          ▒       - 95.01% extract_buf                                                                           ▒            extract_entropy_user                                                                      ▒            urandom_read                                                                              ▒            vfs_read                                                                                  ▒            sys_read                                                                                  ▒            system_call_fastpath                                                                      ▒            __gi___libc_read                                                                          ▒ -   2.06%          python3  [kernel.kallsyms]                           [k] sha_transform            ▒    - sha_transform                                                                                   ▒       - 2.06% extract_buf                                                                            ▒            extract_entropy_user                                                                      ▒            urandom_read                                                                              ▒            vfs_read                                                                                  ▒            sys_read                                                                                  ▒            system_call_fastpath                                                                      ▒            __gi___libc_read                                                                          ▒ -   0.74%          python3  [kernel.kallsyms]                           [k] _mix_pool_bytes          ▒    - _mix_pool_bytes                                                                                 ▒       - 0.74% __mix_pool_bytes                                                                       ▒            extract_buf                                                                               ▒            extract_entropy_user                                                                      ▒            urandom_read                                                                              ▒            vfs_read                                                                                  ▒            sys_read                                                                                  ▒            system_call_fastpath                                                                      ▒            __gi___libc_read                                                                          ▒     0.44%          python3  [kernel.kallsyms]                           [k] extract_buf              ▒     0.15%          python3  python3.4                                   [.] 0x000000000004b055       ▒     0.10%          python3  [kernel.kallsyms]                           [k] memset                   ▒     0.09%          python3  [kernel.kallsyms]                           [k] copy_user_generic_string ▒     0.07%          python3  multiarray.cpython-34m-x86_64-linux-gnu.so  [.] 0x00000000000b4134       ▒     0.06%          python3  [kernel.kallsyms]                           [k] _raw_spin_unlock_irqresto▒     0.06%          python3  python3.4                                   [.] pyeval_evalframeex

it seems if of time spent calling _raw_spin_lock_irqsave. have no idea means, though.

if problem exists in kernel, should narrow down problem using profiler such oprofile or perf.

i.e. run perf record -a -g , read profiling data saved perf data using perf report. see also: linux perf: how interpret , find hotspots.

in case high cpu usage caused competition /dev/urandom -- allows 1 thread read it, multiple python processes doing so.

python module random using initialization. i.e:

$ strace python -c 'import random; while true:     random.random()' open("/dev/urandom", o_rdonly)     = 4 read(4, "\16\36\366\36}"..., 2500) = 2500 close(4)                                   <--- /dev/urandom closed

you may explicitly ask /dev/urandom using os.urandom or systemrandom class. check code dealing random numbers.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more