Category Archives: Linux Stuff

Beowulf Raspberry Pi cluster project @FH Kärnten – Medical Engineering

For our data science and parallel computing projects at Medical Engineering we are working on a 5x Raspberry PI/Asus Tinker Node for our Beowulf cluster

The final Beowulf cluster will have 20 of this nodes with 5×20 Raspberry PI/Asus Tinker.

Currently we have 20 Raspberry PI 3B running at 1400MHz.

One of our next tasks will be the evalution of the Asus Tinker devices as a replacment option for our Raspberry PI 3B.

20170406_113030

Image 1 of 7

Tune Apache-MPM prefork with some Webminstats

A couple of days ago I started to do some optimizations on our server backends. Most of them are Apache-Prefork PHP powered.

One part of doing Apache-Tuning is to play around with the MPM configuration:

<ifmodule prefork.c>
  StartServers       4
  MinSpareServers    3
  MaxSpareServers   10
  ServerLimit      256
  MaxClients       256
  MaxRequestsPerChild  10000
</ifmodule>

The particular configuration setting for Apache can be found in the file /etc/apache/httpd.conf

I’m using Webmin->Sysstats to get some basic data where I can start from.

First I wanted to know the average and total memory usage of the apache worker process(es).

ApacheMemStat
Apache memory consumption (total, per worker)

As you can see after a few days you get a clew about how our worker memory consumption behaves.

Especially the per worker chart is import which gives you a hint for the MaxClients and ServerLimit settings.

The used bash script :

-getApachePerWorkerMem)
   ps -ylC apache2 | awk '{x += $8;y += 1} END {print x/((y-1)*1024)}';;
-getApacheTotalMemPercent)
   sys_mem=`free -m | awk 'NR==2{print $2 }'`
   apache_total_mem=`ps -ylC apache2 | awk '{x += $8;y += 1} END {print x/1024;}'`
   echo "scale=2;$apache_total_mem/($sys_mem/100)" | bc;;

Another important part is the relation of idle and busy workers.

Apache amount and relation of busy/idle workers
Apache amount and relation of busy/idle workers

Here we see that in this particular case we have an average ratio 10/2 (idle/busy) and our max is about (11/7) which gives a balanced amount of idle and busy workers.

The used bash script :

-getApacheBusyWorker)
  echo `wget -q -O - "$URL" --user="$USER" --password=$PASS |grep BusyWorker |cut -d" " -f2`;;
-getApacheIdleWorker)
  echo `wget -q -O - "$URL" --user="$USER" --password=$PASS |grep IdleWorker |cut -d" " -f2`;;

have fun

Mario

How to fix network mount access errors on Intel SS4400-E storage device

For the last couple of years we have been using  an Intel SS4400-E storage device for our internal backup stuff.

Suddenly the main RAID CIFS share stopped working. It seems for some reason nobody could mount this share anymore. All user got an access denied on mount attempts.

So what the hell was going on: The rather simple web interface wasn’t the burner, in fact the system log does not deserve the name Log 🙂

So mostly every RAID in this class is based on an embedded system driven by our friend Linux. This one is no exception.

So we first need the most important part,…. a working telnet or SSH connection. But a first “test” connection to the SS4400 device gave us “connection refused”  as expected.

Luckily after some googling I found a hind that on this devices you can enable the SSH interface via a hidden CGI script.

http://hostname/ssh_controlF.cgi

Now you can login via “root” and your administration password.

By the way, this device comes with the Swiss Army Knife of Embedded Linux BUSYBOX which I also use for my embedded projects. I’m working with Busybox sense 2004 (version 0.94x), ….. great stuff :-).

But it seems, INTEL has some GPL violation with these devices. lol, ..On our bundled CDs we didn’t get any GPL sources from Busybox and Co? This is only mentioned in passing.

In our case a look into the system logs indicates that our RAID share had some file system problems. INTEL uses XFS as the file system on the storage device. Our system log was filled up with error messages of the file access layer.

In our case the solution was really simple. On device startup the file-system reports that there are errors, but then the corrupt partition is mounted and then ,… kabudl……

To fix this you first have to detach the corresponding RAID with “NASdetach”. Now you can unmount the partition and run the XFS repair binary “xfs_repair“.

After 10 – 15 min the repair should be finished.

Reboot the storage device and you should be able to access your share.

have fun

Mario

Fixing error “httpServer.cpp read, write was not declared in this scope” on building vpl_xmlrpc_jail-1.2 under latest Debian

For all which also got this errors,

httpServer.cpp: In member function "std::string HttpJailServer::receive(int, e_t, size_t)":
httpServer.cpp:145:36: error: "read"was not declared in this scope
httpServer.cpp: In member function "void HttpJailServer::sendRaw(const char*,ze_t)":
httpServer.cpp:216:49: error: "write" was not declared in this scope
httpServer.cpp: In member function "void HttpJailServer::send(int, const stri, const string&)":
httpServer.cpp:250:15: error: "close" was not declared in this scope

on building vpl_xmlrpc_jail-1.2 for Moodle VPL module, there seems to be a missing include for these methods.

Just add,

#include <unistd.h>

to the httpServer.cpp, these should fix it.

have fun

Mario

SVN Couldn’t perform atomic initialization fix

After upgrading from debian squeeze I run into the problem that every commit was failing with “Couldn’t perform atomic initialization“.

See Debian bug report

My apache logs are showing :

[error] [client ] Couldn’t perform atomic initialization [500, #200029]

[error] [client ] Couldn’t perform atomic initialization [500, #200029]

[error] [client ] SQLite compiled for 3.7.4, but running with 3.7.3 [500, #200030]

So it looks like that libapache2-svn is build against libsqlite3 3.7.4 which is currently in Sid. In Squezze we have libsqlite3 3.7.3

A quick fix would be to take the libsqlite3 package from Sid for now.

have fun

Apache module_webdav 0600 file mask problem (Ubuntu Lucid)

I’m using WebDAV as our central access point for the users web homes. The running apache is an MPM-ITK. This Version of the apche server uses a special Multi-Processing Module which allows to run each vhost under a separate uid and gid…, cool :-). So you can have quotas and all the other user related stuff in your vhosts.

I forgot the OS is a Ubuntu Lucid LTS, just for info.

A few days ago colleagues told me that the have problems to read/execute files via normal HTTP access which they have uploaded before. A quick look via SSH turned out that the newly created files had all 0600 rights. Because of that the access user/group schema is XYuser/www-data the web server couldn’t access the files anymore.

Hmm, shit, this shouldn’t happen here. So as a brave Linux user, i added umask 022 to the apache environment variable. But,.. what the hell, the files are always created with 0600. ….Oook this will not be a 5min job.

After an hour digging around with google it turns out that this is bug (link) in the recent WebSVN module used by the Lucid distro.

To fix this i downloaded from Debian testing the apache2.2-bin deflated it and copied the corresponding DAV modules to my server. … FIXED 🙂

Remember that taking packages from Debian or another Ubuntu distro is only a temporary solution! 🙂

Have fun

Little Linux server tuning, blacklist not needed modules

For my VM Linux Severs I’m using following kernel module blacklist:

 blacklist mii
 blacklist serio_raw
 blacklist pcspkr
 blacklist psmouse
 blacklist lp
 blacklist parport
 blacklist snd
 blacklist soundcore
 blacklist snd_via82xx
 blacklist gameport
 blacklist snd_ac97_codec
 blacklist parport_pc

Don’t forget to call update-initramfs -u after altering the blacklist.conf !

Do only blacklist the mii module if you have the vmxnet module up and running !!

have fun 🙂

VMware – force the load of the vmxnet module

I noticed that on our virtual environments (VMware ESX) based on  GNU/Linux (mostly Debian or Ubuntu distros ) the vmxnet modul is not used or not correctly loaded. Because of  lsmod shows also a loaded pcnet32 module which uses mii.

I added follwing script snippet to the /etc/init.d/open-vm-tools

 log_progress_msg "vmhgfs"; modprobe vmhgfs
 log_progress_msg "vmsync"; modprobe vmsync
 log_progress_msg "vmxnet - stoping network"; /etc/init.d/networking stop
 log_end_msg 0
 log_progress_msg "vmxnet - removing pcnet32"; rmmod pcnet32
 log_end_msg 0
 log_progress_msg "vmxnet - removing vmxnet"; rmmod vmxnet
 log_end_msg 0
 log_progress_msg "vmxnet - loading vmxnet "; modprobe vmxnet
 log_end_msg 0
 log_progress_msg "vmxnet - starting network";/etc/init.d/networking start
 log_end_msg 0

Check the dmesg out for:

[ 42.273810] vmxnet 0000:00:11.0: PCI INT A disabled
[ 51.808321] VMware vmxnet virtual NIC driver
[ 51.808422] vmxnet 0000:00:11.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[ 51.815601] Found vmxnet/PCI at 0x1424, irq 18.
[ 51.816793] features: ipCsum zeroCopy partialHeaderCopy
[ 51.820787] numRxBuffers = 100, numRxBuffers2 = 1
[ 62.762575] eth0: no IPv6 routers present

This addon to the startup script removes the pcnet32 and vmxnet32 modules and loads explicit vmxnet. 🙂
Take care if you get an symbol not found error while loading the vmghfs module. If so you have to uncomment the loading of this module.

Have fun.

OpenKM our new Document Management System, nice…..

For months I was searching for a system capable to track and store out internal documents. I tested several systems like alfresco, opendocman,…..

In the end I decided to give OpenKM a try.

OpenKM runs on a JBoss application server, so you are bound to Java.  They use Jackrabbit for their hierarchical content storage.  The frontend is realized with GWT (Google Web Tollkit).

For a full feature list click here.

OpenKM install hints:

I installed OpenKM on a recent Debian 6.0

Getting .pdf, .docx, .xlsx preview running:

First you need to install OpenOffice. Make sure you install …-math, …-writer, … -draw, …-calc packages.

OpenOffice hast to bee run in headless mode. I use following startup script:

#!/bin/sh
### BEGIN INIT INFO
# Provides:        OpenOffice Headless
# Required-Start:    
# Required-Stop:    
# Default-Start:    2 3 5
# Default-Stop:        
# Short-Description:    OpenOffice Headless
### END INIT INFO

PIDFILE=/var/run/openoffice-server.pid

case "$1" in
'start')
 if [ -f $PIDFILE ]; then
 echo "OpenOffice headless server has already started."
 sleep 5
 exit
 fi
 echo "Starting OpenOffice headless server"
 soffice "-accept=socket,host=localhost,port=8100;urp;StarOffice.ServiceManager" -nologo -headless -nofirststartwizard & > /dev/null 2>&1
 touch $PIDFILE
 ;;
'stop')
 if [ -f $PIDFILE ]; then
 echo "Stopping OpenOffice headless server."
 killall -9 soffice && killall -9 soffice.bin
 rm -f $PIDFILE
 exit
 fi
 echo "Openoffice headless server is not running."
 exit
 ;;
*)
 echo "Usage: $0 { start | stop }"
 ;;
esac
exit 0

To get the preview stuff working you need also pdf2swf. Check your /bin directory there should be a pdf2swf executable or if you prefer you could take the swftool package from ubuntu.

The last step to get the previewer feature running is to set the corresponding config. flags at the OpenKm.cfg file.

  • system.openoffice=on
  • system.pdf2swf=/usr/bin/pdf2swf

Mail Notification configuration:

Mail notification is straight forward. Go to /opt/OpenKM-4.1_JBoss-4.2.3.GA/server/default/deploy/mail-service.xml . Edit the section to your needs.

If you face the problem that the links in your notification mails are pointing to localhost, you can alter the application.url flag in your OpenKM.cfg.

  • application.url={yourURL}:{yourPort}/OpenKM/es.git.openkm.frontend.Main/index.jsp

Changing from port 8080 -> 80

GoTo /opt/OpenKM-4.1_JBoss-4.2.3.GA/server/default/deploy/jboss-web.deployer/server.xml and edit

Change the port field to 80.

Securing your JBoss

One last thing needs to be done :-). Security! Holy Moly… what… yep.. Jboss runs out of the box like an open barn door. So a good idea would be to secure the jmx-console and web-console stuff. Because of that the jmx-console and web-console are standard servlets you can protect them very easily via enabling the security-constraint.

Securing the jmx-console:

1. Edit \server\default\deploy\jmx-console.war\WEB-INF\web.xml and uncomment the security-constraint section:

<!-- A security constraint that restricts access to the HTML JMX console
 to users with the role JBossAdmin. Edit the roles to what you want and
 uncomment the WEB-INF/jboss-web.xml/security-domain element to enable
 secured access to the HTML JMX console.-->
 <security-constraint>
 <web-resource-collection>
 <web-resource-name>HtmlAdaptor</web-resource-name>
 <description>An example security config that only allows users with the
 role JBossAdmin to access the HTML JMX console web application
 </description>
 <url-pattern>/*</url-pattern>
 <http-method>GET</http-method>
 <http-method>POST</http-method>
 </web-resource-collection>
 <auth-constraint>
 <role-name>JBossAdmin</role-name>
 </auth-constraint>
 </security-constraint>

 <login-config>
 <auth-method>BASIC</auth-method>
 <realm-name>JBoss JMX Console</realm-name>
 </login-config>

 <security-role>
 <role-name>JBossAdmin</role-name>
 </security-role>
</web-app>

2. Edit \server\default\deploy\jmx-console.war\WEB-INF\jboss-web.xml. Uncomment the following section:

 <jboss-web>
 <!-- Uncomment the security-domain to enable security. You will
 need to edit the htmladaptor login configuration to setup the
 login modules used to authentication users.-->
 <security-domain>java:/jaas/jmx-console</security-domain>

</jboss-web>

3. Edit \server\default\conf\props\jmx-console-roles.properties file

4. Edit \server\default\conf\props\jmx-console-users.properties file

5. Edit \server\default\conf\login-config.xml edit following section

 props/web-console-users.properties
 props/web-console-roles.properties

6. under \server\default\conf\props\ copy and rename the two jmx-console-xx to web-console-xx.

Automatically start OpenKM (Jboss) via init scripts:

If you want to start your OpenKM automatically at boot time you can use this init script:

#!/bin/sh
# /etc/init.d/jbossokm: Start and stop JBoss Application Service
### BEGIN INIT INFO
# Provides:        OpenKM
# Required-Start:    
# Required-Stop:    
# Default-Start:    2 3 5
# Default-Stop:        
# Short-Description:    OpenKM
### END INIT INFO

ECHO=/bin/echo
TEST=/usr/bin/test
JBOSS_START_SCRIPT=/opt/OpenKM-4.1_JBoss-4.2.3.GA/bin/run.sh
JBOSS_STOP_SCRIPT=/opt/OpenKM-4.1_JBoss-4.2.3.GA/bin/shutdown.sh

$TEST -x $JBOSS_START_SCRIPT || exit 0
$TEST -x $JBOSS_STOP_SCRIPT || exit 0

start() (
 $ECHO "Starting JBoss OKM."
 rm -rf /path/to/server/default/tmp
 rm -rf /path/to/server/default/work
 rm -rf /path/to/server/default/log
 su -l -c "$JBOSS_START_SCRIPT -b 0.0.0.0 > /dev/null 2> /dev/null &"
 $ECHO "Done."
 )

stop () (
 $ECHO "Stopping JBoss OKM. "
 su -l -c "$JBOSS_STOP_SCRIPT -S > /dev/null &"
 sleep 10
 $ECHO "Done."
)

case "$1" in
 start )
 start
 ;;
 stop )
 stop
 ;;
 restart )
 stop
 sleep 30
 start
 ;;
 * )
 $ECHO "Usage: jbossokm {start|stop|restart}"
 exit 1
 esac
 exit 0

Shutdown problems (network unreachable exception)

If you get an exception at shutdown or your shutdown script does not kill your JBoss then you could check sysctl-setting under /etc/sysctl.d/bindv6only.conf.

Make sure that the net.ipv6.bindv6only=0. !

If you would like to show your appreciation consider to support the guys from OpenKM with a donation 🙂

See PayPal subscription link at the bottom of the page

 


Fun with OpenEMM installation

A few days ago  a colleague asked me if I could install a Newsletter System for him on one of our servers. He decided to use OpenEMM 6.1.

OpenEMM is a feature-rich enterprise software for e-mail marketing, newsletters and service mails (transaction mails and event or time triggered mails). OpenEMM offers sophisticated bounce management, link tracking, lots of realtime statistics, a CMS module and a scripting feature to implement individual tasks…….  cool 🙂

Installation goes by hand, cause sadly there are no deps.

However – after a bit of googling I found a wiki which descriptives the installation process for the common linux distros.

As a brave installation instruction follower I went through the installation step by step.

In the end there was only one task left to start the application server resin and login in for the first time.

And the fun starts 🙂

Instead of a shiny login screen I got….

An error occurred

javax.servlet.ServletException:  org.springframework.jdbc.UncategorizedSQLException: Hibernate  operation: Cannot open connection; uncategorized SQLException for SQL  [???]; SQL state [null]; error code [0]; Cannot create  PoolableConnectionFactory (Communications link failure  The last packet sent successfully to the server was 0 milliseconds ago.  The driver has not received any packets from the server.); nested  exception is org.apache.commons.dbcp.SQLNestedException: Cannot create  PoolableConnectionFactory (Communications link failure  The last packet sent successfully to the server was 0 milliseconds ago.  The driver has not received any packets from the server.)

hmmm, so what is jdbc trying us to say?

So I checked the stack trace:

Caused by: java.net.ConnectException: Connection refused
 at java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
 at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
 at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
 at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
 at java.net.Socket.connect(Socket.java:529)
 at java.net.Socket.connect(Socket.java:478)
 at java.net.Socket.(Socket.java:375)
 at java.net.Socket.(Socket.java:218)
 at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
 at com.mysql.jdbc.MysqlIO.(MysqlIO.java:293)
 ... 57 more

“Connection refused” what the hell…. ok, database credentials are ok, openemm and openemm_cms schema are created. The MySQL error log says nothing? I’m able to connect to the schemas via mysql client ……. oh mannn, it turned out that I’m not the only one with this problem, lol, but all forum posts I could find so far offered no solution for my problem.

Then after half a day more or less searching for a solution I found a post where someone is talking about to check the MySQL “bind-address” parameter.  Checking my.cnf  turns out that the address binding was set to the external address of the server, lol what…, who changed that???

Quickly changed the address-binding back to “localh0st” restarted OpenEMM and requested the logging screen. Unsurprisingly the HTML error stays the same but,…. the stack trace changed…yeahhhh.

Now the trace shows:

Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
 at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2497)
 at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:599)
 ... 72 more

Humpf,…… firing on google. I interpreted this message that jdbc could successfully establish a connection to the database and than for some reason the connection was closed from the db side.

A few hours later checking a lot posts concerning this problem I found a small side node in a post that it make sense to check the host.allow entrys.

Added ….

mysqld : localhost : allow

and ….. it works yeahhhhhh 🙂