CPU usage with 3rd sigs (SOLVED)

This forum is for all copfilter version 2 support related questions in English. (IPCop version 2)
Post Reply
ShelbyGT500
Posts: 846
Joined: 13 May 2010 22:37
Location: FRANCE

CPU usage with 3rd sigs (SOLVED)

Post by ShelbyGT500 »

Hi,

Since I'm testing Copfilter V2 with ICAP , I find that the CPU Usage is very important.

I reported a similar problem with the last release of 3rd sigs on copfilter V1: viewtopic.php?f=3&t=215&start=30#p3494.

I done a try with 3rd sigs disabled, and the CPU usage is now normal:
Image

My questions:
- Are bofhland databases include in last release of Copfilter V2 ?
- If not, is it expected that CPU usage is so important (20 % all the time on my installation with athlon X2 4200 , and 80 % with Celeron ) with 3rd sigs ? That was not the case with copfilter V1, and I thought that 3rd sigs need only 2 gb memory without permanently urging the CPU.

Regards.
Last edited by ShelbyGT500 on 26 Feb 2012 12:55, edited 1 time in total.

karesmakro
Site Admin
Posts: 1280
Joined: 09 Dec 2009 21:17

Re: CPU usage with 3rd sigs

Post by karesmakro »

Can you tell us the process list, which cause this high load?
Perhaps there is a update process running all the time ...

ShelbyGT500
Posts: 846
Joined: 13 May 2010 22:37
Location: FRANCE

Re: CPU usage with 3rd sigs

Post by ShelbyGT500 »

Hi Kare,

I think you're right, it seems there is a problem with 3 rd update:

Here are the logs:

Cron4hourly:
I found there is an update (cron.4 hourly) every 5 minutes:

[CET] 2012-02-25 16:35:07
backing up current databases... done
searching for updates...
/var/log/copfilter/default/opt/tools/bin/cron.4hourly: line 62: /var/log/copfilter/default/opt/tools/bin/wget: No such file or directory
broken download, tmpfile deleted...
updates downloaded...
checking for corrupted databases...chmod: cannot access `/tmp/clamdatabases/*': No such file or directory
chown: cannot access `/tmp/clamdatabases/*': No such file or directory
done...
restart of clamd suspended...
no newer files available. Nothing updated!
restart of clamd suspended...
[CET] 2012-02-25 16:35:07
______________________________________________________________________________________________________________________________
[CET] 2012-02-25 16:40:12
backing up current databases... done
searching for updates...
/var/log/copfilter/default/opt/tools/bin/cron.4hourly: line 62: /var/log/copfilter/default/opt/tools/bin/wget: No such file or directory
broken download, tmpfile deleted...
updates downloaded...
checking for corrupted databases... done...
reloading databases...
clamd running with pid 1428
2 databases correctly updated...
[CET] 2012-02-25 16:40:29
______________________________________________________________________________________________________________________________
[CET] 2012-02-25 16:46:26
backing up current databases... done
searching for updates...
/var/log/copfilter/default/opt/tools/bin/cron.4hourly: line 62: /var/log/copfilter/default/opt/tools/bin/wget: No such file or directory
broken download, tmpfile deleted...
updates downloaded...
checking for corrupted databases... done...
restart of clamd suspended...
no newer files available. Nothing updated!
restart of clamd suspended...
[CET] 2012-02-25 16:47:52
______________________________________________________________________________________________________________________________
[CET] 2012-02-25 16:51:22
backing up current databases... done
searching for updates...
/var/log/copfilter/default/opt/tools/bin/cron.4hourly: line 62: /var/log/copfilter/default/opt/tools/bin/wget: No such file or directory
broken download, tmpfile deleted...
updates downloaded...
checking for corrupted databases... done...
restart of clamd suspended...
no newer files available. Nothing updated!
restart of clamd suspended...
[CET] 2012-02-25 16:52:36
______________________________________________________________________________________________________________________________
[CET] 2012-02-25 16:56:21
backing up current databases... done
searching for updates...
/var/log/copfilter/default/opt/tools/bin/cron.4hourly: line 62: /var/log/copfilter/default/opt/tools/bin/wget: No such file or directory
broken download, tmpfile deleted...
updates downloaded...
checking for corrupted databases... done...
restart of clamd suspended...
no newer files available. Nothing updated!
restart of clamd suspended...
[CET] 2012-02-25 16:57:36
______________________________________________________________________________________________________________________________
[CET] 2012-02-25 17:02:14
backing up current databases... done
searching for updates...
/var/log/copfilter/default/opt/tools/bin/cron.4hourly: line 62: /var/log/copfilter/default/opt/tools/bin/wget: No such file or directory
broken download, tmpfile deleted...
updates downloaded...
checking for corrupted databases... done...
restart of clamd suspended...
no newer files available. Nothing updated!
restart of clamd suspended...
[CET] 2012-02-25 17:05:30


In monit logs:
CET Feb 24 19:38:01] info : Starting monit HTTP server at [*:446]
[CET Feb 24 19:38:01] info : monit HTTP server started
[CET Feb 24 19:38:01] info : 'system_ipcop.localdomain' Monit reloaded
[CET Feb 24 19:38:03] info : Shutting down monit HTTP server
[CET Feb 24 19:38:04] info : monit HTTP server stopped
[CET Feb 24 19:38:04] info : monit daemon with pid [1976] killed
[CET Feb 24 19:38:04] info : 'system_ipcop.localdomain' Monit stopped
[CET Feb 24 19:38:05] info : Starting monit daemon with http interface at [*:446]
[CET Feb 24 19:38:05] info : Starting monit HTTP server at [*:446]
[CET Feb 24 19:38:05] info : monit HTTP server started
[CET Feb 24 19:38:05] info : 'system_ipcop.localdomain' Monit started
[CET Feb 24 19:42:30] info : Awakened by the SIGHUP signal
[CET Feb 24 19:42:30] info : Reinitializing monit - Control file '/var/log/copfilter/default/opt/monit/etc/monitrc'
[CET Feb 24 19:42:30] info : Shutting down monit HTTP server
[CET Feb 24 19:42:30] info : monit HTTP server stopped
[CET Feb 24 19:42:30] info : Starting monit HTTP server at [*:446]
[CET Feb 24 19:42:30] info : monit HTTP server started
[CET Feb 24 19:42:30] info : 'system_ipcop.localdomain' Monit reloaded
[CET Feb 24 19:43:58] info : Awakened by the SIGHUP signal
[CET Feb 24 19:43:58] info : Reinitializing monit - Control file '/var/log/copfilter/default/opt/monit/etc/monitrc'
[CET Feb 24 19:43:58] info : Shutting down monit HTTP server
[CET Feb 24 19:43:59] info : monit HTTP server stopped
[CET Feb 24 19:43:59] info : Starting monit HTTP server at [*:446]
[CET Feb 24 19:43:59] info : monit HTTP server started
[CET Feb 24 19:43:59] info : 'system_ipcop.localdomain' Monit reloaded
[CET Feb 24 19:44:11] info : Awakened by the SIGHUP signal
[CET Feb 24 19:44:11] info : Reinitializing monit - Control file '/var/log/copfilter/default/opt/monit/etc/monitrc'
[CET Feb 24 19:44:11] info : Shutting down monit HTTP server
[CET Feb 24 19:44:11] info : monit HTTP server stopped
[CET Feb 24 19:44:11] info : Starting monit HTTP server at [*:446]
[CET Feb 24 19:44:11] info : monit HTTP server started
[CET Feb 24 19:44:11] info : 'system_ipcop.localdomain' Monit reloaded
[CET Feb 24 19:45:38] info : Awakened by the SIGHUP signal
[CET Feb 24 19:45:38] info : Reinitializing monit - Control file '/var/log/copfilter/default/opt/monit/etc/monitrc'
[CET Feb 24 19:45:38] info : Shutting down monit HTTP server


in 3pmodify log:

added for new use with clamd only:
9 SecuriteInfo databases: securiteinfo.hdb, honeynet.hdb, secinfobat.hdb, secinfodos.hdb, secinfoelf.hdb, secinfohtm.hdb, secinfooff.hdb, secinfopdf.hdb, secinfosh.hdb
1 MalwarePatrol databases: mbl.ndb
[CET] 2012-02-25 16:35:07
_______________________________________________________________________________________________________________________________
added for new use with clamd only:
9 SecuriteInfo databases: securiteinfo.hdb, honeynet.hdb, secinfobat.hdb, secinfodos.hdb, secinfoelf.hdb, secinfohtm.hdb, secinfooff.hdb, secinfopdf.hdb, secinfosh.hdb
1 MalwarePatrol databases: mbl.ndb
[CET] 2012-02-25 16:40:29
_______________________________________________________________________________________________________________________________
added for new use with clamd only:
9 SecuriteInfo databases: securiteinfo.hdb, honeynet.hdb, secinfobat.hdb, secinfodos.hdb, secinfoelf.hdb, secinfohtm.hdb, secinfooff.hdb, secinfopdf.hdb, secinfosh.hdb
1 MalwarePatrol databases: mbl.ndb
[CET] 2012-02-25 16:47:52
______________________________________________________________________________________________________________________________
added for new use with clamd only:
9 SecuriteInfo databases: securiteinfo.hdb, honeynet.hdb, secinfobat.hdb, secinfodos.hdb, secinfoelf.hdb, secinfohtm.hdb, secinfooff.hdb, secinfopdf.hdb, secinfosh.hdb
1 MalwarePatrol databases: mbl.ndb
[CET] 2012-02-25 16:52:36


EDIT : Here is the process list:
root@ipcop:~ # ps -x
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
PID TTY STAT TIME COMMAND
1 ? Ss 0:00 init [3]
2 ? S 0:00 [kthreadd]
3 ? S 0:00 [migration/0]
4 ? S 0:00 [ksoftirqd/0]
5 ? S 0:00 [migration/1]
6 ? S 0:00 [ksoftirqd/1]
7 ? S 0:00 [events/0]
8 ? S 0:00 [events/1]
9 ? S 0:00 [khelper]
14 ? S 0:00 [async/mgr]
58 ? S 0:00 [sync_supers]
60 ? S 0:00 [bdi-default]
62 ? S 0:00 [kblockd/0]
63 ? S 0:00 [kblockd/1]
64 ? S 0:00 [kacpid]
65 ? S 0:00 [kacpi_notify]
66 ? S 0:00 [kacpi_hotplug]
166 ? S 0:00 [kseriod]
197 ? S 0:00 [kconservative/0]
198 ? S 0:00 [kconservative/1]
224 ? S 0:03 [kswapd0]
225 ? S 0:00 [aio/0]
226 ? S 0:00 [aio/1]
555 ? S 0:00 [ksuspend_usbd]
577 ? S 0:00 [khubd]
588 ? S 0:00 [ata/0]
589 ? S 0:00 [ata/1]
590 ? S 0:00 [ata_aux]
602 ? S 0:00 [scsi_eh_0]
603 ? S 0:00 [scsi_eh_1]
606 ? S 0:00 [scsi_eh_2]
608 ? S 0:00 [scsi_eh_3]
671 ? S 0:00 [usbhid_resumer]
715 ? S 0:01 [kjournald]
747 ? S<s 0:00 /sbin/udevd --daemon
827 ? S< 0:00 /sbin/udevd --daemon
828 ? S< 0:00 /sbin/udevd --daemon
934 ? S 0:00 [kjournald]
989 ? Sl 0:00 /usr/sbin/rsyslogd -c4
1002 ? Ss 0:00 /usr/sbin/acpid -n
1031 ? S 0:01 [flush-8:0]
1287 ? Ss 0:00 /usr/sbin/fcron
1289 ? Ss 0:00 /usr/sbin/httpd
1826 ? Ss 0:00 /usr/sbin/dhcpcd --hostname=ipcop wan-1
1831 tty1 Ss+ 0:00 /sbin/agetty --noclear tty1 9600
1832 tty2 Ss+ 0:00 /sbin/agetty tty2 9600
2100 ? Ss 0:00 /usr/sbin/squid -s
2863 ? Ss 0:04 /var/log/copfilter/default/opt/mail-spamassassin/defa
5357 ? Ssl 0:00 /var/log/copfilter/default/opt/monit/default/bin/moni
6770 ? Ss 0:00 sshd: root@pts/0
6773 pts/0 Ss 0:00 -bash
6790 ? S 0:00 /usr/sbin/fcron
6791 ? Ss 0:00 /bin/bash -c /usr/local/bin/copfilter_3pcron >> /tmp/
6795 ? S 0:00 /bin/sh /usr/local/bin/copfilter_3pcron
6803 ? S 0:00 /bin/bash /var/log/copfilter/default/opt/tools/bin/3r
7645 ? S 0:00 /bin/sh /var/log/copfilter/default/opt/tools/bin/cron
7822 ? S 0:00 /bin/sh /var/log/copfilter/default/opt/tools/bin/cron
7833 ? R 0:02 /var/log/copfilter/default/opt/clamav/default/bin/cla
7834 pts/0 R+ 0:00 ps -x
15395 ? Ss 0:00 /usr/sbin/sshd
root@ipcop:~ #


I've installed IPCOP and Copfilter recently. And the CPU Usage was important with my previous installation.

Regards.

FischerM
Site Admin
Posts: 545
Joined: 09 Dec 2009 19:24
Location: Rheinbach

Re: CPU usage with 3rd sigs

Post by FischerM »

Hi!
ShelbyGT500 wrote:line 62: /var/log/copfilter/default/opt/tools/bin/wget: No such file or directory
'wget' seems to be missing.

Open console and enter:

Code: Select all

ln -s /usr/bin/wget /var/log/copfilter/default/opt/tools/bin/wget
HTH
Matthias

karesmakro
Site Admin
Posts: 1280
Joined: 09 Dec 2009 21:17

Re: CPU usage with 3rd sigs

Post by karesmakro »

Because wget is now a standard addon since IPCop 2.0.3, I removed it from our package.
But I missed to copy the modified scripts, which was using wget, to the new copfilter version

:cry:

ShelbyGT500
Posts: 846
Joined: 13 May 2010 22:37
Location: FRANCE

Re: CPU usage with 3rd sigs

Post by ShelbyGT500 »

Hi Matthias and Kare,

Thank you for your answer.
FischerM wrote:

Code: Select all

ln -s /usr/bin/wget /var/log/copfilter/default/opt/tools/bin/wget

Done.

This seems to have solved my problem: :D :D
- the CPU usage is now normal :D
- the 3rd sigs update is ok: :D

Cron.4hourly logs:
[CET] 2012-02-26 09:58:56
backing up current databases... done
searching for updates...
/var/log/copfilter/default/opt/tools/bin/cron.4hourly: line 62: /var/log/copfilter/default/opt/tools/bin/wget: No such file or directory
broken download, tmpfile deleted...
updates downloaded...
checking for corrupted databases... done...
restart of clamd suspended...
no newer files available. Nothing updated!
restart of clamd suspended...
[CET] 2012-02-26 10:00:11
______________________________________________________________________________________________________________________________
[CET] 2012-02-26 10:07:00
backing up current databases... done
searching for updates...
--2012-02-26 10:07:02-- http://www.malware.com.br/cgi/submit?ac ... clamav_ext
Resolving http://www.malware.com.br... 72.14.190.204
Connecting to http://www.malware.com.br|72.14.190.204|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: `STDOUT'
0K .......... .......... .......... .......... .......... 59.6K
50K .......... .......... .......... .......... .......... 198K
100K .......... .......... .......... .......... .......... 225K
150K .......... .......... .......... .......... .......... 227K
200K .......... .......... .......... .......... .......... 312K
250K .......... .......... .. 74.8K=2.0s
2012-02-26 10:07:05 (137 KB/s) - written to stdout [278765]
updates downloaded...
Checking for corrupted databases... done...
reloading databases...
clamd running with pid 3829
1 databases correctly updated...
[CET] 2012-02-26 10:08:51


Moreover, it also seemed to solve a problem of update for the blacklist of C-ICAP. :D

One last thing: Have you an idea to solve a similar problem with copfilter V1? viewtopic.php?f=3&t=215&start=30#p3494

Thank you in advance.

Regards.

karesmakro
Site Admin
Posts: 1280
Joined: 09 Dec 2009 21:17

Re: CPU usage with 3rd sigs (SOLVED)

Post by karesmakro »

ShelbyGT500 wrote:Moreover, it also seemed to solve a problem of update for the blacklist of C-ICAP.
The C-ICAP update should run anyway, because there was not used a fix path!
The cron.daily and cron4.hourly were the only ones which had to be modified.

regards

ShelbyGT500
Posts: 846
Joined: 13 May 2010 22:37
Location: FRANCE

Re: CPU usage with 3rd sigs (SOLVED)

Post by ShelbyGT500 »

Hi Kare,
karesmakro wrote:The C-ICAP update should run anyway,
Maybe it was a coincidence, but this is what I noticed.

I reinstalled copfilter to confirm this problem. You're right, because there is no update for c-icap on my test machine , after adding symlink for '/usr/bin/wget.
I will write a new topic for this issue.

EDIT:
Matthias, i noticed you completed the wiki very quickly ;)

Regards.

Post Reply