Anyone watching the servers? Does not look like it...

[dcabbar]

Amir and Techs,

I have commented on before about server load issues on mykonos many times and opened tickets intermittenly.

And, after all of these, got the classic "everything is fine now" BS... So, let me ask you:

1. How hard is it really to write a simple cronjob script that monitors server load and on repetative abuses disable the abbusive accounts and/or trigger a notification to admins for manual review?

2. I have pasted "most abusive" job list before a few times, and it keeps getting ignored...

But, once again:

I wrote a very simple script (literally, only 105 lines of perl code), which runs every 5 minutes. It looks at current job list and top results, correlates them and figures out heaviest users at that particular time. And, it is quite apparent that:

taau
hotteenz
bestdom
ossnzj
fluddeni

users run cronjobs or serve pages which keep taking a lot of CPU... These users keep coming up in the "most abusive process" list all the time and CPU load is shown as 11 - 20 all the time.

Server is really not usable like this and very unstable due these load issues (which has been the case for months now!).

Are you planning to do anything about this? Or, will I get yet another BS about 'everything is fine now'...

Please advise...

PS:

Here is the list for the last 30 minutes:

09/23/2007 17:30:03	6.58	8.18	8.27	7.67	[fluddeni:perl -T -I /home/fluddeni/public_html/twbw-test/bin /home/fluddeni/public_html/twbw-test/bin/statistics,fluddeni:perl -T -I /home/fluddeni/public_html/twiki/bin /home/fluddeni/public_html/twiki/bin/statistics]

 =======> [ 23796 fluddeni  20   0  8180 4500 1420 R    9  0.1   0:00.08 perl -T -I /home/fl
            perl -T -I /home/fluddeni/public_html/twiki/bin /home/fluddeni/public_html/twiki/bin/statistics]

 =======> [ 23727 fluddeni  20   0  4748 3332 1224 R    6  0.1   0:00.05 perl -T -I /home/fl
            perl -T -I /home/fluddeni/public_html/twbw-test/bin /home/fluddeni/public_html/twbw-test/bin/statistics]

--------------------------------------------------------------------

09/23/2007 17:31:03	7.27	7.98	8.19	7.81	[europlak:/usr/lib/courier-imap/bin/imapd /home/europlak/mail/europlakusa.com/jairo,hotteenz:/usr/bin/php4-cgi r,ossnzj:/usr/bin/php4-cgi f,taau:/usr/bin/php4-cgi s]

 =======> [ 26843 taau      18   0 88632  11m 6600 R   21  0.3   0:00.22 /usr/bin/php4-cgi s ]
 =======> [ 26847 ossnzj    18   0 87332  10m 6896 R   18  0.3   0:00.13 /usr/bin/php4-cgi f ]
 =======> [ 26852 hotteenz  18   0 51360 5408 4172 R    6  0.1   0:00.04 /usr/bin/php4-cgi r ]
 =======> [ 26457 europlak  16   0  3184 1144  704 S    3  0.0   0:00.54 /usr/lib/courier-im
            /usr/lib/courier-imap/bin/imapd /home/europlak/mail/europlakusa.com/jairo]

--------------------------------------------------------------------

09/23/2007 17:32:03	12.88	9.43	8.67	10.32	[hotteenz:/usr/bin/php4-cgi r]
 =======> [ 27823 hotteenz  18   0 86168 8076 5312 R    4  0.2   0:00.05 /usr/bin/php4-cgi r ]

--------------------------------------------------------------------

09/23/2007 17:33:07	9.68	9.12	8.61	9.13	[driggers:/usr/bin/php4-cgi i,hotteenz:/usr/bin/php4-cgi r,taau:/usr/bin/php4-cgi m,taau:/usr/bin/php4-cgi showthread.php]
 =======> [ 28637 taau      18   0 86408  13m  10m R   39  0.3   0:00.33 /usr/bin/php4-cgi m ]
 =======> [ 28673 taau      17   0 88100  11m 6600 R   18  0.3   0:00.12 /usr/bin/php4-cgi s
            /usr/bin/php4-cgi showthread.php]
 =======> [ 28675 driggers  18   0 86196 7880 5072 R   12  0.2   0:00.06 /usr/bin/php4-cgi i ]
 =======> [ 28671 hotteenz  18   0 86168  33m  30m R   10  0.9   0:00.10 /usr/bin/php4-cgi r ]

--------------------------------------------------------------------

09/23/2007 17:34:02	9.25	9.10	8.63	8.99	[driggers:/usr/bin/php4-cgi index2.php,hotteenz:/usr/bin/php4-cgi o,taau:/usr/bin/php4-cgi s]
 =======> [ 29344 taau      17   0 89200  12m 7364 R   80  0.3   0:01.00 /usr/bin/php4-cgi s ]
 =======> [ 29357 hotteenz  18   0 86304 7776 4928 R   12  0.2   0:00.06 /usr/bin/php4-cgi o ]
 =======> [ 29362 driggers  18   0 86332 7892 5072 R   12  0.2   0:00.06 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index2.php]

--------------------------------------------------------------------

09/23/2007 17:35:02	7.61	8.59	8.48	8.22

--------------------------------------------------------------------

09/23/2007 17:36:02	7.98	8.53	8.46	8.32	[a2domain:/usr/bin/php4-cgi index.php]
 =======> [ 30654 a2domain  18   0 51228 4736 3764 R    8  0.1   0:00.04 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index.php]

--------------------------------------------------------------------

09/23/2007 17:37:02	5.74	7.80	8.22	7.25	[taau:/usr/bin/php4-cgi forumdisplay.php]
 =======> [ 31261 taau      18   0 88512  11m 6700 R   80  0.3   0:00.44 /usr/bin/php4-cgi f
            /usr/bin/php4-cgi forumdisplay.php]

--------------------------------------------------------------------

09/23/2007 17:38:02	11.72	8.82	8.52	9.68	[bestdom:/usr/bin/php4-cgi i,driggers:/usr/bin/php4-cgi index2.php,hotteenz:/usr/bin/php4-cgi f,hotteenz:/usr/bin/php4-cgi fontimage.php]
 =======> [ 31942 bestdom   16   0 54228  11m 7984 D   25  0.3   0:00.19 /usr/bin/php4-cgi i ]
 =======> [ 31954 hotteenz  18   0 86320  24m  21m R   20  0.6   0:00.10 /usr/bin/php4-cgi f ]
 =======> [ 31952 driggers  15   0 86332 7940 5092 S   12  0.2   0:00.06 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index2.php]
 =======> [ 31955 hotteenz  18   0 86176 7360 4592 R   10  0.2   0:00.05 /usr/bin/php4-cgi f
            /usr/bin/php4-cgi fontimage.php]

--------------------------------------------------------------------

09/23/2007 17:39:07	14.21	9.90	8.90	11	[driggers:/usr/bin/php4-cgi index2.php,hotteenz:/usr/bin/php4-cgi r,taau:/usr/bin/php4-cgi i]
 =======> [ 341 hotteenz  17   0 53400 7504 4736 R   21  0.2   0:00.11 /usr/bin/php4-cgi r ]
 =======> [ 343 driggers  18   0 86308 7840 5028 R   10  0.2   0:00.05 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index2.php]
 =======> [ 32694 taau      18   0 86440  23m  20m R    8  0.6   0:00.11 /usr/bin/php4-cgi i ]

--------------------------------------------------------------------

09/23/2007 17:40:04	23.34	13.31	10.13	15.59	[a2domain:/usr/bin/php4-cgi i,driggers:/usr/bin/php4-cgi index2.php,hotteenz:/usr/bin/php4-cgi r,hotteenz:/usr/bin/php4-cgi r,taau:/usr/bin/php4-cgi i]
 =======> [ 1199 taau      18   0 86504  38m  35m R   19  1.0   0:00.13 /usr/bin/php4-cgi i ]
 =======> [ 1205 hotteenz  18   0 86172  34m  31m R   19  0.9   0:00.10 /usr/bin/php4-cgi r ]
 =======> [ 1130 a2domain  18   0 86480  15m  12m R   17  0.4   0:00.23 /usr/bin/php4-cgi i ]
 =======> [ 1221 driggers  18   0 86324 7888 5072 R   12  0.2   0:00.06 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index2.php]
 =======> [ 1220 hotteenz  18   0 51364 5088 3952 R    8  0.1   0:00.04 /usr/bin/php4-cgi r ]

--------------------------------------------------------------------

09/23/2007 17:41:29	34.61	18.63	12.21	21.81	[ossnzj:/usr/bin/php4-cgi s,taau:/usr/bin/php4-cgi s,taau:[php4-cg] <defunct>]
 =======> [ 3784 taau      16   0 90228  13m 7464 S    6  0.4   0:00.70 /usr/bin/php4-cgi s ]
 =======> [ 3791 taau      18   0     0    0    0 Z    3  0.0   0:00.36 [php4-cg] <defunct> ]
 =======> [ 3778 ossnzj    16   0 88760  12m 7524 S    2  0.3   0:00.28 /usr/bin/php4-cgi s ]

--------------------------------------------------------------------

09/23/2007 17:42:17	42.46	23.57	14.27	26.76	[taau:/usr/bin/php4-cgi articles.php,taau:/usr/bin/php4-cgi f,taau:/usr/bin/php4-cgi f]
 =======> [ 4852 taau      18   0 88412  12m 8068 R    4  0.3   0:00.48 /usr/bin/php4-cgi f ]
 =======> [ 4913 taau      17   0 88508  11m 6672 R    3  0.3   0:00.30 /usr/bin/php4-cgi f ]
 =======> [ 4883 taau      18   0 86648 8116 4860 S    1  0.2   0:00.07 /usr/bin/php4-cgi a
            /usr/bin/php4-cgi articles.php]

--------------------------------------------------------------------

09/23/2007 17:43:04	25.49	21.98	14.23	20.56	[a2domain:/usr/bin/php4-cgi index.php,driggers:/usr/bin/php4-cgi i,hotteenz:/usr/bin/php4-cgi r,taau:/usr/bin/php4-cgi i,taau:[php4-cgi] <defunct>]
 =======> [ 6353 taau      18   0 88420  11m 6448 R   37  0.3   0:00.19 /usr/bin/php4-cgi f
            [php4-cgi] <defunct>]
 =======> [ 6356 taau      18   0 86444  34m  31m R   23  0.9   0:00.12 /usr/bin/php4-cgi i ]
 =======> [ 6350 a2domain  18   0 92812  13m 4880 D   19  0.4   0:00.10 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index.php]
 =======> [ 6351 hotteenz  18   0 86168  33m  30m R   19  0.9   0:00.10 /usr/bin/php4-cgi r ]
 =======> [ 6349 driggers  15   0 86328 7940 5092 R   12  0.2   0:00.06 /usr/bin/php4-cgi i ]

--------------------------------------------------------------------

09/23/2007 17:44:04	19.94	20.87	14.32	18.37	[a2domain:/usr/bin/php4-cgi index.php,hotteenz:/usr/bin/php4-cgi r,hotteenz:/usr/bin/php4-cgi r,taau:/usr/bin/php4-cgi attachment.php,taau:/usr/bin/php4-cgi showthread.php]
 =======> [ 7264 hotteenz  18   0 86172  35m  32m R   21  0.9   0:00.11 /usr/bin/php4-cgi r ]
 =======> [ 7274 taau      18   0 88404  11m 6324 R   17  0.3   0:00.22 /usr/bin/php4-cgi a
            /usr/bin/php4-cgi attachment.php]
 =======> [ 7293 a2domain  18   0 92812  13m 4880 R    8  0.4   0:00.10 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index.php]
 =======> [ 7311 hotteenz  17   0 18072 2384 1984 R    4  0.1   0:00.02 /usr/bin/php4-cgi r ]
 =======> [ 7278 taau      18   0 18072 2244 1916 R    2  0.1   0:00.01 /usr/bin/php4-cgi s
            /usr/bin/php4-cgi showthread.php]

--------------------------------------------------------------------

09/23/2007 17:45:03	23.25	21.74	15.03	20	[driggers:/usr/bin/php4-cgi index2.php,gowsocce:/usr/bin/php4-cgi index.php,gowsocce:convert -density 15,taau:/usr/bin/php4-cgi showthread.php,tormaca:/usr/bin/php4-cgi account.php]
 =======> [ 8344 taau      18   0 89332  13m 7404 R   66  0.3   0:00.41 /usr/bin/php4-cgi s
            /usr/bin/php4-cgi showthread.php]
 =======> [ 8346 tormaca   18   0 51704 6156 4556 S    4  0.2   0:00.05 /usr/bin/php4-cgi a
            /usr/bin/php4-cgi account.php]
 =======> [ 8347 driggers  16   0 86328 7928 5092 S    4  0.2   0:00.06 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index2.php]
 =======> [ 8377 gowsocce  18   0 14108 6840 3916 R    4  0.2   0:00.02 convert -density 15 ]
 =======> [ 32315 gowsocce  15   0  102m  31m  10m S    4  0.8   0:10.47 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index.php]

--------------------------------------------------------------------

09/23/2007 17:46:03	15.75	20.02	14.87	16.87	[bestdom:/usr/bin/php4-cgi categ-tree.php,hotteenz:/usr/bin/php4-cgi index.php,hotteenz:/usr/bin/php4-cgi index.php,hotteenz:/usr/bin/php4-cgi index.php,hotteenz:/usr/bin/php4-cgi r,hotteenz:/usr/bin/php4-cgi randomFreePages.php]
 =======> [ 9409 hotteenz  18   0 86168  38m  36m R   21  1.0   0:00.11 /usr/bin/php4-cgi r
            /usr/bin/php4-cgi randomFreePages.php]
 =======> [ 9414 hotteenz  17   0 51496 5496 4224 R    8  0.1   0:00.04 /usr/bin/php4-cgi r ]
 =======> [ 9411 hotteenz  18   0 18072 2608 2076 R    6  0.1   0:00.03 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index.php]
 =======> [ 9413 hotteenz  18   0 18072 2380 1984 R    4  0.1   0:00.02 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index.php]
 =======> [ 9415 bestdom   17   0 18072 2608 2080 R    4  0.1   0:00.02 /usr/bin/php4-cgi c
            /usr/bin/php4-cgi categ-tree.php]
 =======> [ 9157 hotteenz  15   0 86860 8632 5312 S    2  0.2   0:00.13 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index.php]

--------------------------------------------------------------------

09/23/2007 17:47:12	15.96	18.99	14.84	16.59	[bestdom:/usr/bin/php4-cgi c,hotteenz:/usr/bin/php4-cgi i,hotteenz:/usr/bin/php4-cgi r,ossnzj:[php4-cg] <defunct>,taau:/usr/bin/php4-cgi e,taau:/usr/bin/php4-cgi showthread.php]
 =======> [ 10604 hotteenz  15   0 88656  13m 8672 S   32  0.4   0:00.37 /usr/bin/php4-cgi i ]
 =======> [ 10647 hotteenz  18   0 86168  23m  21m R   11  0.6   0:00.09 /usr/bin/php4-cgi r ]
 =======> [ 10628 bestdom   15   0 86592 9828 6676 S   10  0.3   0:00.08 /usr/bin/php4-cgi c ]
 =======> [ 10637 taau      17   0 86300 9192 6300 S    9  0.2   0:00.07 /usr/bin/php4-cgi s
            /usr/bin/php4-cgi showthread.php]
 =======> [ 10573 ossnzj    18   0     0    0    0 Z    7  0.0   0:00.30 [php4-cg] <defunct> ]
 =======> [ 10513 taau      15   0 88728  13m 8084 S    2  0.3   0:00.31 /usr/bin/php4-cgi e ]

--------------------------------------------------------------------

09/23/2007 17:48:02	11.78	17.42	14.53	14.57	[hotteenz:/usr/bin/php4-cgi index.php]
 =======> [ 11418 hotteenz  15   0 87000  11m 8496 S   21  0.3   0:00.11 /usr/bin/php4-cgi i
            /usr/bin/php4-cgi index.php]

[GhostRider2110]

Woud you share the script with us? I would like to see exactly what you are collecting. Very rusty in perl and it would help my learning curve... Also are you taking any metrix on IO wait time during these slowdowns... And yes I agree there are slowdowns... My simple logging of load shows already this morning the load jumped above 20...

See-ya
Mitch

[dcabbar]

No response from the admins... Interesting

Here is the code:

#!/usr/local/bin/perl

unlink("/tmp/tmp_load_982.txt");
unlink("/tmp/tmp_load_982.txt.2");
system("export DISPLAY=132; /usr/bin/top -bc -n 1| head -20 > /tmp/tmp_load_982.txt");
system("ps -e -o pid,command > /tmp/tmp_load_982.txt.2");

%procLookup = ();

open(IN, "/tmp/tmp_load_982.txt.2");
while ($line = <IN>) {
  chop($line);
  if ($line =~ /([^\s]+)\s+(.+)/) {
    $pid = $1;
    $cmd = $2;
    $procLookup{$pid} = $cmd;
  }
}
close(IN);

open(IN, "/tmp/tmp_load_982.txt");
$tmpS  = "";
$tmpS2 = "";
$lineFound = "";
$processStarted = 0;
$avg = 0;
@processList = ();
$cTime = getCurrentTime();

while ($line = <IN>) {
  chop($line);
  if ($line =~ /load average:[ \t]+([0-9]+\.[0-9]+),[ \t]+([0-9]+\.[0-9]+),[ \t]+([0-9]+\.[0-9]+)/) {
    $avg = int(($1 + $2 + $3)/ 0.03) / 100.0;
    $tmpS = $cTime . "\t" . $1 . "\t" . $2 . "\t" . $3 . "\t" . $avg;
    $lineFound = "[$line]";
  } elsif (($line =~ /\s+PID\s+USER\s+PR\s+NI/) && ($avg > 7)) {
    $processStarted = 1;
  } elsif ($processStarted) {
    if ($line =~ /([^\s]+)\s+([^\s]+)\s+[^\s]+\s+[^\s]+\s+[^\s]+\s+[^\s]+\s+[^\s]+\s+[^\s]+\s+[^\s]+\s+[^\s]+\s+[^\s]+\s+(.+)/) {
      $pid     = $1;
      $user    = $2;
      $detailFound = 0;
      if (!($procLookup{$pid})) {
        $process = $3;
      } else {
        $process = $procLookup{$pid};
        $detailFound = 1;
      }

      if (($user eq "root") || ($user eq "__USER_NAME__") ||
          ($user eq "nobody") || ($user eq "postgres") ||
          ($user eq "tomcat") || ($user eq "mailnull") ||
          ($user eq "cpanel") || ($user eq "") ||
          ($user eq "mysql") || ($user eq "mailman")) {
        next;
      }
 
      $line = trim($line); $process = trim($process);
      if ($detailFound) {
        $tmpS2 .= " =======> [ $line\n            $process]\n";
      } else {
        $tmpS2 .= " =======> [ $line ]\n";
      }
      $data = trim($user . ":" . $process);
      push (@processList, $data);
    }
  }
}
close(IN);
if ($tmpS2 ne "") {
  $tt = time();
  $ffName = "/home/__USER_NAME__/tmp/load_list_" . $tt . ".txt";
  system("cp /tmp/tmp_load_982.txt $ffName");
}
unlink("/tmp/tmp_load_982.txt");
unlink("/tmp/tmp_load_982.txt.2");

if ($tmpS) {
  $data = join(",", sort @processList);
  $tmpS .= ($data ? "\t[$data]" : "");
  open(OUT, ">>/home/__USER_NAME_/public_html/cpu.txt");
  print OUT $tmpS . "\n" . $tmpS2;
  close(OUT);
}

sub getCurrentTime() {
  ($second, $minute, $hour, $dayOfMonth, $month, 
     $yearOffset, $dayOfWeek, $dayOfYear, $daylightSavings) = localtime();
  $year = 1900 + $yearOffset;
  $month++;
  return fillString($month, 2) . "/" . fillString($dayOfMonth, 2) . "/" . "$year " . fillString($hour, 2) . ":" . fillString($minute, 2) . ":" . fillString($second, 2);
}

sub fillString() {
  my($string, $length) = @_;
  return ("0" x ($length - length($string))) . $string;
}

sub trim {
  my($string) = @_;
  $string =~ s/^\s+//;
  $string =~ s/\s+$//;
  return $string;
}

Just replace __USER_NAME__ with your username...

[GhostRider2110]

Thanks going to play with the code a bit... learn something....

Just looking at the load avg over the last 24 hours.. there were at least 20 spikes where the load was over 10...

There was a big spike around 1825:
18:25:01 up 102 days, 13:41, 1 user, load average: 14.21, 8.06, 8.37
18:30:02 up 102 days, 13:46, 1 user, load average: 28.53, 16.19, 11.29
18:35:01 up 102 days, 13:51, 1 user, load average: 10.29, 12.97, 11.30
18:40:03 up 102 days, 13:56, 1 user, load average: 42.91, 24.47, 15.94
18:45:17 up 102 days, 14:01, 2 users, load average: 12.61, 15.59, 14.50
18:50:02 up 102 days, 14:06, 1 user, load average: 8.80, 10.21, 12.50
18:55:01 up 102 days, 14:11, 1 user, load average: 5.41, 7.39, 10.66

See-ya

[Elik]

Hello,

We do have server monitoring in place for all servers, and it is reported to our IRC Channel by our monitoring bot to give real time status on any servers that hit beyond 10 every 3 minutes. But the IRC Channel is not open for anyone since it is used internally for all techs on duty.

[dcabbar]

Hi Elik,

Thanks for the update, but I can see from the output of my script that mykonos keeps getting above 10 almost on a regular basis, and it is always same users, which makes me think that nobody is doing anything about it?

Are you taking any action on these by any chance? And, if so, why do I keep seeing these overloading issues?

Thanks.

[Elik]

The issue is that most of those overload issues occurs in the morning while I am offshift, so I was not able to monitor it. I have instructed the staff to take snapshop for half hour intervals to see if we can find the common cause of which user is responsible for it.

It have been taken since I have suspended one account for it, and the loads dropped down for it after reviews.

[GhostRider2110]

There was a big spike today between 12-1 system time:
12:05:01 up 104 days, 7:20, 1 user, load average: 10.78, 7.08, 6.26
12:10:01 up 104 days, 7:25, 1 user, load average: 14.18, 10.31, 7.84
12:15:01 up 104 days, 7:30, 1 user, load average: 7.16, 10.42, 8.80
12:20:06 up 104 days, 7:35, 1 user, load average: 19.77, 13.49, 10.31
12:25:01 up 104 days, 7:40, 1 user, load average: 19.13, 14.30, 11.28
12:30:02 up 104 days, 7:45, 1 user, load average: 4.30, 9.80, 10.39
12:35:01 up 104 days, 7:50, 1 user, load average: 20.77, 16.95, 13.02
12:40:01 up 104 days, 7:55, 1 user, load average: 8.57, 11.28, 11.63
12:45:01 up 104 days, 8:00, 1 user, load average: 7.89, 9.01, 10.56
12:50:05 up 104 days, 8:05, 1 user, load average: 10.34, 7.93, 9.53

[dcabbar]

Well,

I see many overloaded time slices, and it is the same users again: bestdom, taau, fluddeni, thrccor

For example the last spike is below:

09/26/2007 19:01:02 12.53 7.41 5.46 8.46 [bestdom:/usr/bin/php4-cgi index.php]
=======> [ 5033 bestdom 18 0 86724 10m 7104 S 18 0.3 0:00.09 /usr/bin/php4-cgi i
/usr/bin/php4-cgi index.php]
09/26/2007 19:02:02 9.44 7.44 5.59 7.49 [andaman:/usr/bin/php4-cgi content.php,bestdom:/usr/bin/php4-cgi index.php,bestdom:/usr/bin/php4-cgi index.php,bestdom:/usr/bin/php4-cgi index.php,bestdom:/usr/bin/php4-cgi index.php,fmuqlyyr:/usr/bin/php4-cgi index.php]
=======> [ 5385 bestdom 18 0 86724 10m 7156 S 16 0.3 0:00.08 /usr/bin/php4-cgi i
/usr/bin/php4-cgi index.php]
=======> [ 5378 andaman 18 0 86900 8384 5076 S 12 0.2 0:00.09 /usr/bin/php4-cgi c
/usr/bin/php4-cgi content.php]
=======> [ 5380 bestdom 17 0 86856 10m 7100 S 12 0.3 0:00.09 /usr/bin/php4-cgi i
/usr/bin/php4-cgi index.php]
=======> [ 5370 fmuqlyyr 17 0 86572 10m 7540 R 8 0.3 0:00.08 /usr/bin/php4-cgi i
/usr/bin/php4-cgi index.php]
=======> [ 5379 bestdom 15 0 86856 10m 7100 S 8 0.3 0:00.08 /usr/bin/php4-cgi i
/usr/bin/php4-cgi index.php]
=======> [ 5369 bestdom 17 0 86720 10m 7088 S 4 0.3 0:00.08 /usr/bin/php4-cgi i
/usr/bin/php4-cgi index.php]

[GhostRider2110]

What exactly do you mean by "overloaded timeslices"?

Well thrccor is one of my sites, which is a small web site for our local R/C club. which has standard HTML... Nothing hardcore about it. ALso has a small flash chat room which uses a mysql DB. Which will timeout during heavy load... Which also shows the flaw in looking just at time slices for fairness of system... By default the system will give avaible CPU/MEMORY to programs requesting it... There is no easy way to "partition" off a system and make it efficient ( in this configuration.. dedicated servers or using virtualization such as VMware is a different story). Also, i have not had time to look at your script close but if you added in looking at I/O wait time, I bet you will see that is the major factor in the delays on the system. Just because a single process for thrccor has 22% of the CPU for 2 seconds does not mean it is "overloading timeslices"

[dcabbar]

IO Waits being the major factor in delays? What do you mean? We are talking about CPU overload here...

Furthermore, instant spikes and looking at them without the global context don't make sense, I agree.

But, this script logs these processes (and users running them) ONLY WHEN load average is greater than 7.

And, if you see the same users over and over and over during these load spikes, that tells you that these users are abusing the system (CPU mainly). For example, this script has been running for a while now, and almost all the time I see these 3 users (and even more, but I can't remember right now): hotteenz, bestdom, taau

And, the worst part is, System Admins don't monitor the system at all, they don't know this, they don't do anything about this. Which, I believe, makes Micfo an unstable place to host your "serious" sites on. I am OK with this, but these fluctuations take it to a level that you can't even use it for personal stuff!

[GhostRider2110]

First.. I am not saying that there is not a problem. There is overloading on the system. It is affecting some web sites... More those which make use of MySQL I believe. Since my web site I have on there well still respond well even when the load on the system is 10-15. The chat program running is affected more do to the use of MySQL and the IO waiting causing a timeout.

Your script does show a consistent set of users who's web sites seem to be very active. Most of them look like forum use from the processes being shown.

I am saying.. it is not JUST CPU overload. A load of 10 on a dual proc system may not be 100% CPU related and may not bring a system to a halt. I have about a two dozen Linux servers I manage which many will average a load of 10-20 but run smooth. Your script is giving a snapshot of the current process running when the load is at a certain point... What brought that load to that point? Since the load avg includes running, runnable and blocked processes.. I/O plays a big roll in that avg.

Back to the busy web sites on the server, maybe it is time they are moved to a VPS or dedicated server. I also know that I would not put a business critical web server on a shared hosting service, no mater what the SLA is.

[Elik]

Dcabbar,

I been watching the server on the periods that I am around and I do not see any issues with it. Your comments shows that you do not have much experience on how the system works, and I would suggest you stop complaining til you have more knowledge on how the systems works.

Lot of those issues are more of IO stress, not due to CPU overload. I have servers that do go and run around 15 to 20, and they are fine because they are more of IO traffic than the memory or CPU usage.

Anyway, I consider this matter closed since I do not see much issues with them at this time, and the loads that is on the server are acceptable when I checked on them. There going to be a major overhaul coming soon, which we will be going Apache 2 on all systems, which there will be much more efficient usage of resources. But it is still under development and with my work with Cpanel, it is coming along nicely. It is nearly there, but not there yet.

[dcabbar]

Elik,

With all due respect, I feel bad that someone like you, who have been failing to do your job for months now (LOOK AT ALL THE COMPLAINTS ON THIS FORUM), is trying to tell who knows what. I am even more worried now than ever... In fact, the state that these machines have been in for months is a good proof of your level of expertise!...

Furthermore, I don't think you are in a place to judge my expertise and/or knowledge, and on top of everything, you don't have any right to talk to a customer with this tone, especially while I am stating a very simple fact!... But, leaving aside personal attacks...

Please read some unix system admin books. Sun Performance and Tuning book defines load average as:

The load average is the sum of the run queue length and the number of jobs currently running on the CPUs.

Although it has some relation to IOWAIT, Load Average is not directly related to IOWAIT (as quote above states very clearly). And, I have no clue how you can say:

Lot of those issues are more of IO stress, not due to CPU overload. I have servers that do go and run around 15 to 20, and they are fine because they are more of IO traffic than the memory or CPU usage.

First of all, ask anyone on any forum or any expert, they will tell you that 15-20 is not an acceptable load. Servers can go fine (whatever fine is according to you!), but response times will suck (due to cpu slicing and process management issues)! And, Io vs Memory Utilization vs CPU load have difference impacts and again although they are somewhat related, they usually are caused by different problems.

When a process goes into IOWAIT, it does not consume CPU, it is not considered as RUNNABLE, unless it is waiting for a disk resource. Hence, they are usually not the cause of the Load Average!... Now, if you are having Disk IO issues, that is a whole difference issue and it is a clear indicator of system admin incompetence, since there are many ways of resolving this issue, such as increasing RAM, separating out MySQL to RAID-5/RAID-10 machines and configuring them properly and putting tons of memory, etc. etc.

I been watching the server on the periods that I am around

This is clearly what has been going on, and this is exactly what I have problems with! If you are sick, can't come to work, or have anything else, what does that mean? Are you saying that the whole system and hundreds (possibly thousands) of customers are at your mercy? What kind of an operations management is this? These servers have been in trouble for months, and you were the person in charge. What kind of watching is that?


Last but not the least, I don't have root priv on this machine, hence I have access to limited set of tools. Therefore, I have no way of knowing what exactly is going on. But, what I know is that, if system load is constantly shooting upto 15-20 and even more, this means:

1. There is too much CPU load on that system, hence the system is not running properly.
2. System Admins are not doing their jobs: they are either putting too many accounts, or not monitoring the system at all, or not optimizing it right.
3. You can't rely on this hosting company at all

And, what I know is that, if you keep seeing processes from same users everytime there is a load spike, that gives you a good idea about where to start hunting the problems!... But, apparently, you are not capable of that.

Anyways, this thread and your comments have clearly demonstrated the real cause of the problems we are having here at Micfo, and I won't even argue in this thread, especially with someone like you anymore.

PS: I hope you don't delete this message. If you think you have the right to post a message like that, you should be able to keep the response!...

[Elik]

Hello,

OKay. Now the old business with the old facility have been done with, which I can focus on the new issues here. I been working on the Apache2, but this going to be new topic for now. I have been talking to management about this problem and here is what we going to do.

We will provision new servers, since the two servers, mykonos and stockholm are underpowered and not of the same specs as the standard servers we use for our customers. However, the downside of this is that it going to require new set of IPs and some little downtime due to shifts of the accounts from old to the new servers.

The new servers will be on the same platform as our reseller/shared servers, which is based on Quad Core Xeons based on Core 2 instead of Dual Core P4 Netburst, and with RAID1 filesystem. This will resolve all the issues and complaints here since we do not have any complaints or issues from other customers who is on the other servers compared to those two.