SLES 11 SP2 High-CPU usage on migration/X threads

Discussion:

F***@bg-phoenics.de

2013-01-23 10:41:52 UTC

Hi Guys,

I have some trouble on my SLES 11 SP2 machines. The machines are installed
on a VMWare ESXi (latest stable version).

machine:~ # ps aux | grep migration
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6 93.0 0.0 0 0 ?
S 2012 93550:09 [migration/0]
root 8 96.5 0.0 0 0 ?
S 2012 97016:21 [migration/1]
root 13 94.8 0.0 0 0 ?
S 2012 95365:44 [migration/2]
root 17 95.6 0.0 0 0 ?
S 2012 96182:33 [migration/3]

"top" shows the same stats (CPU Time+), but %CPU is 0.

Linux lb40016 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012
(d73692b) x86_64 x86_64 x86_64 GNU/inux

Nothing interesting in the logfiles (/var/log/messages, etc). The machine
was not vMotioned. It runs some hours, then the migration threads going to
this point. The actual vmware tools are installed.

after execute "ps aux | grep migration"

machine:~ # mpstat -P ALL
Linux 3.0.13-0.27-default 01/23/13 _x86_64_

11:36:23 CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:36:23 all 0.33 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63
11:36:23 0 0.30 0.00 0.05 0.01 0.00 0.00 0.00
0.00 99.64
11:36:23 1 0.31 0.01 0.03 0.00 0.00 0.00 0.00
0.00 99.65
11:36:23 2 0.37 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.59
11:36:23 3 0.34 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63

machine:~ # sar -u 2 5
Linux 3.0.13-0.27-default 01/23/13 _x86_64_

11:38:17 CPU %user %nice %system %iowait %steal
%idle
11:38:19 all 0.25 0.00 0.13 0.00 0.00
99.62
11:38:21 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:23 all 0.00 0.00 0.13 0.00 0.00
99.87
11:38:25 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:27 all 0.00 0.00 0.13 0.00 0.00
99.87
Average: all 0.05 0.00 0.08 0.00 0.00
99.87

machine:~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 20156 0 0 0 IO-APIC-edge timer
1: 8 0 0 0 IO-APIC-edge i8042
3: 0 1 0 0 IO-APIC-edge
4: 0 0 0 1 IO-APIC-edge
6: 5 0 0 0 IO-APIC-edge floppy
7: 0 0 0 0 IO-APIC-edge parport0
8: 17 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 137 0 0 0 IO-APIC-edge i8042
14: 0 0 0 0 IO-APIC-edge ata_piix
15: 1767460 2377356 45 2282 IO-APIC-edge ata_piix
17: 1866 1300755 21061 2746 IO-APIC-fasteoi ioc0
18: 9141485 0 0 11 IO-APIC-fasteoi eth0
40: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
41: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
42: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
43: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
44: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
45: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
46: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
47: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
48: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
49: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
50: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
51: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
52: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
53: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
54: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
55: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
56: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
57: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
58: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
59: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
60: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
61: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
62: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
63: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
64: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
65: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
66: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
67: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
68: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
69: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
70: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
71: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
72: 11 0 0 0 PCI-MSI-edge vmci
73: 0 0 0 0 PCI-MSI-edge vmci
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 52661861 53572304 61909856 60410143 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring
interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 6948002 7854897 5876118 5547225 Rescheduling interrupts
CAL: 855295 1999 136878 138016 Function call
interrupts
TLB: 178002 356505 228928 210113 TLB shootdowns
TRM: 0 0 0 0 Thermal event
interrupts
THR: 0 0 0 0 Threshold APIC
interrupts
MCE: 0 0 0 0 Machine check
exceptions
MCP: 20107 20107 20107 20107 Machine check polls
ERR: 0
MIS: 0

Does anyone have an idea?

Franz Kinader
______________________________________

BG-Phoenics GmbH
Abteilung Betrieb
Loristraße 6 a
80335 München

Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19

www.bg-phoenics.de
______________________________________
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
Geschäftsführer: Burkhard Wolf (Vorsitz), Walter Lerch

Максим Ткаченко

2013-01-23 10:56:50 UTC

Permalink

Hi!
may be help you this
echo 0 > /proc/sys/kernel/sched_cpulimit_nr_balance
?

Post by F***@bg-phoenics.de
Hi Guys,
I have some trouble on my SLES 11 SP2 machines. The machines are installed
on a VMWare ESXi (latest stable version).
machine:~ # ps aux | grep migration
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6 93.0 0.0 0 0 ?
S 2012 93550:09 [migration/0]
root 8 96.5 0.0 0 0 ?
S 2012 97016:21 [migration/1]
root 13 94.8 0.0 0 0 ?
S 2012 95365:44 [migration/2]
root 17 95.6 0.0 0 0 ?
S 2012 96182:33 [migration/3]
"top" shows the same stats (CPU Time+), but %CPU is 0.
Linux lb40016 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012
(d73692b) x86_64 x86_64 x86_64 GNU/inux
Nothing interesting in the logfiles (/var/log/messages, etc). The machine
was not vMotioned. It runs some hours, then the migration threads going to
this point. The actual vmware tools are installed.
after execute "ps aux | grep migration"
machine:~ # mpstat -P ALL
Linux 3.0.13-0.27-default 01/23/13 _x86_64_
11:36:23 CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:36:23 all 0.33 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63
11:36:23 0 0.30 0.00 0.05 0.01 0.00 0.00 0.00
0.00 99.64
11:36:23 1 0.31 0.01 0.03 0.00 0.00 0.00 0.00
0.00 99.65
11:36:23 2 0.37 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.59
11:36:23 3 0.34 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63
machine:~ # sar -u 2 5
Linux 3.0.13-0.27-default 01/23/13 _x86_64_
11:38:17 CPU %user %nice %system %iowait %steal
%idle
11:38:19 all 0.25 0.00 0.13 0.00 0.00
99.62
11:38:21 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:23 all 0.00 0.00 0.13 0.00 0.00
99.87
11:38:25 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:27 all 0.00 0.00 0.13 0.00 0.00
99.87
Average: all 0.05 0.00 0.08 0.00 0.00
99.87
machine:~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 20156 0 0 0 IO-APIC-edge timer
1: 8 0 0 0 IO-APIC-edge i8042
3: 0 1 0 0 IO-APIC-edge
4: 0 0 0 1 IO-APIC-edge
6: 5 0 0 0 IO-APIC-edge floppy
7: 0 0 0 0 IO-APIC-edge parport0
8: 17 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 137 0 0 0 IO-APIC-edge i8042
14: 0 0 0 0 IO-APIC-edge ata_piix
15: 1767460 2377356 45 2282 IO-APIC-edge ata_piix
17: 1866 1300755 21061 2746 IO-APIC-fasteoi ioc0
18: 9141485 0 0 11 IO-APIC-fasteoi eth0
40: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
41: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
42: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
43: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
44: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
45: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
46: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
47: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
48: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
49: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
50: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
51: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
52: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
53: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
54: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
55: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
56: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
57: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
58: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
59: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
60: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
61: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
62: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
63: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
64: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
65: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
66: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
67: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
68: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
69: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
70: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
71: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
72: 11 0 0 0 PCI-MSI-edge vmci
73: 0 0 0 0 PCI-MSI-edge vmci
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 52661861 53572304 61909856 60410143 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring
interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 6948002 7854897 5876118 5547225 Rescheduling interrupts
CAL: 855295 1999 136878 138016 Function call interrupts
TLB: 178002 356505 228928 210113 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC
interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 20107 20107 20107 20107 Machine check polls
ERR: 0
MIS: 0
Does anyone have an idea?
Franz Kinader
______________________________________
BG-Phoenics GmbH
Abteilung Betrieb
LoristraÃe 6 a
80335 MÃŒnchen
Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19
www.bg-phoenics.de
______________________________________
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
GeschÃ€ftsfÃŒhrer: Burkhard Wolf (Vorsitz), Walter Lerch
_______________________________________________
suse-sles-e mailing list
http://listx.novell.com/mailman/listinfo/suse-sles-e

Максим Ткаченко

2013-01-23 11:03:44 UTC

Permalink

Sorry/ this workaround not wokr on SLES11sp2 :(

may be this help for you;
https://bugzilla.openvz.org/show_bug.cgi?id=1954

Post by ÐÐ°ÐºÑÐ¸Ð¼ Ð¢ÐºÐ°ÑÐµÐ½ÐºÐ¾
Hi!
may be help you this
echo 0 > /proc/sys/kernel/sched_cpulimit_nr_balance
?

Post by F***@bg-phoenics.de
Hi Guys,
I have some trouble on my SLES 11 SP2 machines. The machines are
installed on a VMWare ESXi (latest stable version).
machine:~ # ps aux | grep migration
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6 93.0 0.0 0 0 ?
S 2012 93550:09 [migration/0]
root 8 96.5 0.0 0 0 ?
S 2012 97016:21 [migration/1]
root 13 94.8 0.0 0 0 ?
S 2012 95365:44 [migration/2]
root 17 95.6 0.0 0 0 ?
S 2012 96182:33 [migration/3]
"top" shows the same stats (CPU Time+), but %CPU is 0.
Linux lb40016 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012
(d73692b) x86_64 x86_64 x86_64 GNU/inux
Nothing interesting in the logfiles (/var/log/messages, etc). The machine
was not vMotioned. It runs some hours, then the migration threads going to
this point. The actual vmware tools are installed.
after execute "ps aux | grep migration"
machine:~ # mpstat -P ALL
Linux 3.0.13-0.27-default 01/23/13 _x86_64_
11:36:23 CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:36:23 all 0.33 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63
11:36:23 0 0.30 0.00 0.05 0.01 0.00 0.00 0.00
0.00 99.64
11:36:23 1 0.31 0.01 0.03 0.00 0.00 0.00 0.00
0.00 99.65
11:36:23 2 0.37 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.59
11:36:23 3 0.34 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63
machine:~ # sar -u 2 5
Linux 3.0.13-0.27-default 01/23/13 _x86_64_
11:38:17 CPU %user %nice %system %iowait %steal
%idle
11:38:19 all 0.25 0.00 0.13 0.00 0.00
99.62
11:38:21 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:23 all 0.00 0.00 0.13 0.00 0.00
99.87
11:38:25 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:27 all 0.00 0.00 0.13 0.00 0.00
99.87
Average: all 0.05 0.00 0.08 0.00 0.00
99.87
machine:~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 20156 0 0 0 IO-APIC-edge timer
1: 8 0 0 0 IO-APIC-edge i8042
3: 0 1 0 0 IO-APIC-edge
4: 0 0 0 1 IO-APIC-edge
6: 5 0 0 0 IO-APIC-edge floppy
7: 0 0 0 0 IO-APIC-edge parport0
8: 17 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 137 0 0 0 IO-APIC-edge i8042
14: 0 0 0 0 IO-APIC-edge ata_piix
15: 1767460 2377356 45 2282 IO-APIC-edge ata_piix
17: 1866 1300755 21061 2746 IO-APIC-fasteoi ioc0
18: 9141485 0 0 11 IO-APIC-fasteoi eth0
40: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
41: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
42: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
43: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
44: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
45: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
46: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
47: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
48: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
49: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
50: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
51: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
52: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
53: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
54: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
55: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
56: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
57: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
58: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
59: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
60: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
61: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
62: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
63: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
64: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
65: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
66: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
67: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
68: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
69: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
70: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
71: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
72: 11 0 0 0 PCI-MSI-edge vmci
73: 0 0 0 0 PCI-MSI-edge vmci
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 52661861 53572304 61909856 60410143 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring
interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 6948002 7854897 5876118 5547225 Rescheduling interrupts
CAL: 855295 1999 136878 138016 Function call interrupts
TLB: 178002 356505 228928 210113 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC
interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 20107 20107 20107 20107 Machine check polls
ERR: 0
MIS: 0
Does anyone have an idea?
Franz Kinader
______________________________________
BG-Phoenics GmbH
Abteilung Betrieb
LoristraÃe 6 a
80335 MÃŒnchen
Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19
www.bg-phoenics.de
______________________________________
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
GeschÃ€ftsfÃŒhrer: Burkhard Wolf (Vorsitz), Walter Lerch
_______________________________________________
suse-sles-e mailing list
http://listx.novell.com/mailman/listinfo/suse-sles-e

F***@bg-phoenics.de

2013-01-23 13:29:48 UTC

Permalink

Okay, now this is very strange.

I run following script every minute as a cronjob:

---- CUT ---
#!/bin/bash
echo "== Start $(date)" >> /root/highcpu/log.txt
ps aux | grep migration | grep -v grep >> /root/highcpu/log.txt
ps aux | grep plone | grep -v grep >> /root/highcpu/log.txt

echo "" >> /root/highcpu/log.txt
ps aux >> /root/highcpu/log.txt
echo "== END" >> /root/highcpu/log.txt
echo "" >> /root/highcpu/log.txt
--- CUT ---

With following result:

== Start Wed Jan 23 13:15:01 CET 2013
ps aux | grep migration
root 6 0.0 0.0 0 0 ? S Jan22 0:00
[migration/0]
root 8 0.0 0.0 0 0 ? S Jan22 0:00
[migration/1]
root 13 0.0 0.0 0 0 ? S Jan22 0:00
[migration/2]
root 17 0.0 0.0 0 0 ? S Jan22 0:00
[migration/3]

ps aux (cutted log)
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6 0.0 0.0 0 0 ? S Jan22 0:00
[migration/0]
root 8 0.0 0.0 0 0 ? S Jan22 0:00
[migration/1]
root 13 99.7 0.0 0 0 ? S Jan22 1274:16
[migration/2]
root 17 0.0 0.0 0 0 ? S Jan22 0:00
[migration/3]
== END

In one second, the process migration jumps from 0:00 CPU usage to 1274:16.
The uptime of the server is 1278 minutes. Regarding the Uptime and the CPU
Time the usage ist 99,7% in ps aux, because cpu usage ist not a real time
stat.

Best Regards,
Franz Kinader
______________________________________

BG-Phoenics GmbH
Abteilung Betrieb
Loristraße 6 a
80335 München

Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19

www.bg-phoenics.de
______________________________________
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
Geschäftsführer: Burkhard Wolf (Vorsitz), Walter Lerch

F***@bg-phoenics.de

2013-01-23 11:11:02 UTC

Permalink

Hi,

:-(

Nope, this does not help, saw a lot of other kernel fixes/bugs:

https://bugs.gentoo.org/show_bug.cgi?id=394487
http://git.kernel.org/?p=linux/kernel/git/tip/tip.git;a=commitdiff;h=8f6189684eb4e85e6c593cd710693f09c944450a
http://marc.info/?l=linux-kernel&m=134400163902035&w=2

But this doesn't help, I am restricted to the SLES Pools from Novell and a
vanilla kernel is not an option.
Next interesting step: This affect happens only on VMWare machines. s390x
and real hardware are not affected.

Thanks for your investigations.

______________________________________

BG-Phoenics GmbH
Abteilung Betrieb
LoristraÃe 6 a
80335 MÃŒnchen

Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19

www.bg-phoenics.de
______________________________________
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
GeschÃ€ftsfÃŒhrer: Burkhard Wolf (Vorsitz), Walter Lerch

Von: ÐÐ°ÐºÑÐžÐŒ Ð¢ÐºÐ°ÑÐµÐœÐºÐŸ <***@gmail.com>
An: ***@bg-phoenics.de
Kopie: suse-sles-***@listx.novell.com
Datum: 23.01.2013 12:03
Betreff: Re: [suse-sles-e] SLES 11 SP2 High-CPU usage on
migration/X threads

Sorry/ this workaround not wokr on SLES11sp2 :(

may be this help for you;
https://bugzilla.openvz.org/show_bug.cgi?id=1954

2013/1/23 ÐÐ°ÐºÑÐžÐŒ Ð¢ÐºÐ°ÑÐµÐœÐºÐŸ <***@gmail.com>
Hi!
may be help you this
echo 0 > /proc/sys/kernel/sched_cpulimit_nr_balance
?

2013/1/23 <***@bg-phoenics.de>
Hi Guys,

I have some trouble on my SLES 11 SP2 machines. The machines are installed
on a VMWare ESXi (latest stable version).

machine:~ # ps aux | grep migration
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6 93.0 0.0 0 0 ?
S 2012 93550:09 [migration/0]
root 8 96.5 0.0 0 0 ?
S 2012 97016:21 [migration/1]
root 13 94.8 0.0 0 0 ?
S 2012 95365:44 [migration/2]
root 17 95.6 0.0 0 0 ?
S 2012 96182:33 [migration/3]

"top" shows the same stats (CPU Time+), but %CPU is 0.

Linux lb40016 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012
(d73692b) x86_64 x86_64 x86_64 GNU/inux

Nothing interesting in the logfiles (/var/log/messages, etc). The machine
was not vMotioned. It runs some hours, then the migration threads going to
this point. The actual vmware tools are installed.

after execute "ps aux | grep migration"

machine:~ # mpstat -P ALL
Linux 3.0.13-0.27-default 01/23/13 _x86_64_

11:36:23 CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:36:23 all 0.33 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63
11:36:23 0 0.30 0.00 0.05 0.01 0.00 0.00 0.00
0.00 99.64
11:36:23 1 0.31 0.01 0.03 0.00 0.00 0.00 0.00
0.00 99.65
11:36:23 2 0.37 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.59
11:36:23 3 0.34 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63

machine:~ # sar -u 2 5
Linux 3.0.13-0.27-default 01/23/13 _x86_64_

11:38:17 CPU %user %nice %system %iowait %steal
%idle
11:38:19 all 0.25 0.00 0.13 0.00 0.00
99.62
11:38:21 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:23 all 0.00 0.00 0.13 0.00 0.00
99.87
11:38:25 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:27 all 0.00 0.00 0.13 0.00 0.00
99.87
Average: all 0.05 0.00 0.08 0.00 0.00
99.87

machine:~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 20156 0 0 0 IO-APIC-edge timer

1: 8 0 0 0 IO-APIC-edge i8042

3: 0 1 0 0 IO-APIC-edge
4: 0 0 0 1 IO-APIC-edge
6: 5 0 0 0 IO-APIC-edge
floppy
7: 0 0 0 0 IO-APIC-edge
parport0
8: 17 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 137 0 0 0 IO-APIC-edge i8042

14: 0 0 0 0 IO-APIC-edge
ata_piix
15: 1767460 2377356 45 2282 IO-APIC-edge
ata_piix
17: 1866 1300755 21061 2746 IO-APIC-fasteoi ioc0
18: 9141485 0 0 11 IO-APIC-fasteoi eth0
40: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
41: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
42: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
43: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
44: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
45: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
46: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
47: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
48: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
49: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
50: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
51: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
52: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
53: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
54: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
55: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
56: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
57: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
58: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
59: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
60: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
61: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
62: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
63: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
64: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
65: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
66: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
67: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
68: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
69: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
70: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
71: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
72: 11 0 0 0 PCI-MSI-edge vmci
73: 0 0 0 0 PCI-MSI-edge vmci
NMI: 0 0 0 0 Non-maskable interrupts

LOC: 52661861 53572304 61909856 60410143 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring
interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 6948002 7854897 5876118 5547225 Rescheduling interrupts

CAL: 855295 1999 136878 138016 Function call
interrupts
TLB: 178002 356505 228928 210113 TLB shootdowns
TRM: 0 0 0 0 Thermal event
interrupts
THR: 0 0 0 0 Threshold APIC
interrupts
MCE: 0 0 0 0 Machine check
exceptions
MCP: 20107 20107 20107 20107 Machine check polls
ERR: 0
MIS: 0

Does anyone have an idea?

Franz Kinader
______________________________________

BG-Phoenics GmbH
Abteilung Betrieb
LoristraÃe 6 a
80335 MÃŒnchen

Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19

www.bg-phoenics.de
______________________________________
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
GeschÃ€ftsfÃŒhrer: Burkhard Wolf (Vorsitz), Walter Lerch

_______________________________________________
suse-sles-e mailing list
suse-sles-***@listx.novell.com
http://listx.novell.com/mailman/listinfo/suse-sles-e

--
Ð¡ ÑÐ²Ð°Ð¶ÐµÐœÐžÐµÐŒ,
Ð¢ÐºÐ°ÑÐµÐœÐºÐŸ ÐÐ°ÐºÑÐžÐŒ

--
Ð¡ ÑÐ²Ð°Ð¶ÐµÐœÐžÐµÐŒ,
Ð¢ÐºÐ°ÑÐµÐœÐºÐŸ ÐÐ°ÐºÑÐžÐŒ

F***@bg-phoenics.de

2013-01-23 11:02:11 UTC

Permalink

Hi Maxim,

I dont have this file, should I create it?

-rw-r--r-- 1 root root 0 Jan 23 11:59
/proc/sys/kernel/sched_cfs_bandwidth_slice_us
-rw-r--r-- 1 root root 0 Jan 23 11:59
/proc/sys/kernel/sched_child_runs_first
-rw-r--r-- 1 root root 0 Nov 14 16:05 /proc/sys/kernel/sched_compat_yield
-rw-r--r-- 1 root root 0 Jan 23 11:59 /proc/sys/kernel/sched_latency_ns
-rw-r--r-- 1 root root 0 Jan 23 11:59
/proc/sys/kernel/sched_migration_cost
-rw-r--r-- 1 root root 0 Jan 23 11:59
/proc/sys/kernel/sched_min_granularity_ns
-rw-r--r-- 1 root root 0 Jan 23 11:59 /proc/sys/kernel/sched_nr_migrate
-rw-r--r-- 1 root root 0 Jan 23 11:59 /proc/sys/kernel/sched_rt_period_us
-rw-r--r-- 1 root root 0 Jan 23 11:59 /proc/sys/kernel/sched_rt_runtime_us
-rw-r--r-- 1 root root 0 Jan 23 11:59 /proc/sys/kernel/sched_shares_window
-rw-r--r-- 1 root root 0 Jan 23 11:59 /proc/sys/kernel/sched_time_avg
-rw-r--r-- 1 root root 0 Jan 23 11:59
/proc/sys/kernel/sched_tunable_scaling
-rw-r--r-- 1 root root 0 Jan 23 11:59
/proc/sys/kernel/sched_wakeup_granularity_ns

What does this command do?

I have 4 CPUs configured from the ESXi.

Franz Kinader
______________________________________

BG-Phoenics GmbH
Abteilung Betrieb
LoristraÃe 6 a
80335 MÃŒnchen

Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19

www.bg-phoenics.de
______________________________________
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
GeschÃ€ftsfÃŒhrer: Burkhard Wolf (Vorsitz), Walter Lerch

Von: ÐÐ°ÐºÑÐžÐŒ Ð¢ÐºÐ°ÑÐµÐœÐºÐŸ <***@gmail.com>
An: ***@bg-phoenics.de
Kopie: suse-sles-***@listx.novell.com
Datum: 23.01.2013 11:56
Betreff: Re: [suse-sles-e] SLES 11 SP2 High-CPU usage on
migration/X threads

Hi!
may be help you this
echo 0 > /proc/sys/kernel/sched_cpulimit_nr_balance
?

2013/1/23 <***@bg-phoenics.de>
Hi Guys,

I have some trouble on my SLES 11 SP2 machines. The machines are installed
on a VMWare ESXi (latest stable version).

machine:~ # ps aux | grep migration
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6 93.0 0.0 0 0 ?
S 2012 93550:09 [migration/0]
root 8 96.5 0.0 0 0 ?
S 2012 97016:21 [migration/1]
root 13 94.8 0.0 0 0 ?
S 2012 95365:44 [migration/2]
root 17 95.6 0.0 0 0 ?
S 2012 96182:33 [migration/3]

"top" shows the same stats (CPU Time+), but %CPU is 0.

Linux lb40016 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012
(d73692b) x86_64 x86_64 x86_64 GNU/inux

Nothing interesting in the logfiles (/var/log/messages, etc). The machine
was not vMotioned. It runs some hours, then the migration threads going to
this point. The actual vmware tools are installed.

after execute "ps aux | grep migration"

machine:~ # mpstat -P ALL
Linux 3.0.13-0.27-default 01/23/13 _x86_64_

11:36:23 CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:36:23 all 0.33 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63
11:36:23 0 0.30 0.00 0.05 0.01 0.00 0.00 0.00
0.00 99.64
11:36:23 1 0.31 0.01 0.03 0.00 0.00 0.00 0.00
0.00 99.65
11:36:23 2 0.37 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.59
11:36:23 3 0.34 0.00 0.03 0.00 0.00 0.00 0.00
0.00 99.63

machine:~ # sar -u 2 5
Linux 3.0.13-0.27-default 01/23/13 _x86_64_

11:38:17 CPU %user %nice %system %iowait %steal
%idle
11:38:19 all 0.25 0.00 0.13 0.00 0.00
99.62
11:38:21 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:23 all 0.00 0.00 0.13 0.00 0.00
99.87
11:38:25 all 0.00 0.00 0.00 0.00 0.00
100.00
11:38:27 all 0.00 0.00 0.13 0.00 0.00
99.87
Average: all 0.05 0.00 0.08 0.00 0.00
99.87

machine:~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 20156 0 0 0 IO-APIC-edge timer

1: 8 0 0 0 IO-APIC-edge i8042

3: 0 1 0 0 IO-APIC-edge
4: 0 0 0 1 IO-APIC-edge
6: 5 0 0 0 IO-APIC-edge
floppy
7: 0 0 0 0 IO-APIC-edge
parport0
8: 17 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 137 0 0 0 IO-APIC-edge i8042

14: 0 0 0 0 IO-APIC-edge
ata_piix
15: 1767460 2377356 45 2282 IO-APIC-edge
ata_piix
17: 1866 1300755 21061 2746 IO-APIC-fasteoi ioc0
18: 9141485 0 0 11 IO-APIC-fasteoi eth0
40: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
41: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
42: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
43: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
44: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
45: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
46: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
47: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
48: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
49: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
50: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
51: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
52: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
53: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
54: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
55: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
56: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
57: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
58: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
59: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
60: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
61: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
62: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
63: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
64: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
65: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
66: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
67: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
68: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
69: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
70: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
71: 0 0 0 0 PCI-MSI-edge PCIe
PME, pciehp
72: 11 0 0 0 PCI-MSI-edge vmci
73: 0 0 0 0 PCI-MSI-edge vmci
NMI: 0 0 0 0 Non-maskable interrupts

LOC: 52661861 53572304 61909856 60410143 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring
interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 6948002 7854897 5876118 5547225 Rescheduling interrupts

CAL: 855295 1999 136878 138016 Function call
interrupts
TLB: 178002 356505 228928 210113 TLB shootdowns
TRM: 0 0 0 0 Thermal event
interrupts
THR: 0 0 0 0 Threshold APIC
interrupts
MCE: 0 0 0 0 Machine check
exceptions
MCP: 20107 20107 20107 20107 Machine check polls
ERR: 0
MIS: 0

Does anyone have an idea?

Franz Kinader
______________________________________

BG-Phoenics GmbH
Abteilung Betrieb
LoristraÃe 6 a
80335 MÃŒnchen

Fon: +49 (0) 89-12179-92 48
Fax: +49 (0) 89-12179-9 99
Mobil: +49 (0) 173-618-14 19

www.bg-phoenics.de
______________________________________
Sitz der Gesellschaft: Hannover
Handelsregistergericht: Amtsgericht Hannover
HRB Nr.: 59345
GeschÃ€ftsfÃŒhrer: Burkhard Wolf (Vorsitz), Walter Lerch

_______________________________________________
suse-sles-e mailing list
suse-sles-***@listx.novell.com
http://listx.novell.com/mailman/listinfo/suse-sles-e

--
Ð¡ ÑÐ²Ð°Ð¶ÐµÐœÐžÐµÐŒ,
Ð¢ÐºÐ°ÑÐµÐœÐºÐŸ ÐÐ°ÐºÑÐžÐŒ