Java7
Shell
What does GREP(1) in manual of grep mean?
Nov 28th
I came across this question and was in my mind thinking about what “1″ GREP(1) enclosed in the brackets means.
Apparently grep(1) means grep is a command line.
There are altogether eight categories of man pages:
- General Commands
- System Calls
- Subroutines
- Special Files
- File Formats
- Games
- Macros and Conventions
- Maintenence Commands
Therefore GREP(1) means GREP is a general command. As simple as that!
Cheers.
Write shell scripts that scan through your application logs regardless of log rotation cycle
Nov 24th
We faced a particularly interesting problem today, one of our shell scripts designed to pick up keywords from application logs (logs generated by our online order shop – a J2EE web application – not JavaEE yet :P) failed intermittently because the application log rotated earlier than the next scheduled run of the shell script.
The shell script is a simple implementation – it scans through a particular directory to pick up log file ocESB.log, it then continues to look(using grep) for keywords like NullPointerException, CorbaException, it then sends an email notification when it finds them.
The application log rotates on a size basis, when it hits 1 MB, it will rotate, i.e. the file name will change.
Our application log:
(Note that osESB.log will always rotate once it reaches 1 MB)
-rwx------+ 1 chinboon Domain Users 1008565 Nov 23 05:06 ocESB.log -rwx------+ 1 chinboon Domain Users 1000547 Nov 23 11:03 ocESB.log.20111123_110321 -rwx------+ 1 chinboon Domain Users 1012174 Nov 23 11:13 ocESB.log.20111123_111320 -rwx------+ 1 chinboon Domain Users 1002229 Nov 23 11:23 ocESB.log.20111123_112341 -rwx------+ 1 chinboon Domain Users 1000322 Nov 23 11:35 ocESB.log.20111123_113513
Existing Shell Script
(It checks for NullPointerException in the log file ocESB.log)
#!/bin/bash
set -x
LOGFILE="/apps/dell/logs/ocESB.log"
EMAIL="support@dell.com"
errorCount=`grep -c "NullPointerException" ${LOGFILE}`
if [ $errorCount -ne '0' ]
then
mail -s "PRD: Error found in ESB logs!" $EMAIL
fi
Background
- Web application logs to file ocESB.log, this file is size rotated, currently, set at 1 MB.
- The size rotated application log rolls approximately every 30 minutes or so during business hours. The log rotation is slower during off peak hours, which could be anything between one hour and a few hours (especially during the midnight where no one is purchasing or browsing the online web store)
- Another shell script batch job runs on the application log ocESB.log, this shell script is a cron job that is scheduled to run once every 30 minutes.
Problem
- Application logs that rotate faster than the next scheduled shell script eludes the shell script.
- Application logs containing keywords that are slow to rotate (especially during off peak hours) will get picked up by the batch again and again, giving false alarms to the user.
Limitation
- Application logging is a controlled behavior that follows a strict application release and change management life-cycle, also it takes too many red tapes to get this setting changed.
- The shell script may be changed at anytime, however user confirmed that they had only wanted this script to notify them once every 30 minutes. Only when keywords are picked up, an email notification is sent.
So how do we go about solving this? Yes, many ways, such as keeping a separate file to keep track of logs files that have been scanned .. But here is one eye opening solution.
Solution
- Create a check file, a check/touch file is a blank file that is meant to be empty, its sole purpose is for us to compare the time stamp. (Software engineering best practice would probably call for you to name this check/touch file with a .check postfix, and with a suffix same as your shell script name for maintainability. For example, if your shell script is called EWS.sh, then name your check/touch file EWS.check)
- Modify your existing cron batch script to search for logs based on the time stamp of the check/touch file created above.
Source Code
(Changes in bold)
#!/bin/bash
set -x
LOG_PATH="/apps/dell/logs/"
LAST_CHECK="EWS.check"
EMAIL="support@dell.com"
find $LOG_PATH -name "ocESB.log*" -type "f" -newer "$LAST_CHECK" | while read file
do
errorCount=`grep -c "NullPointerException" ${LOGFILE}`
if [ $errorCount -ne '0' ]
then
mail -s "PRD: Error found in ESB logs!" $EMAIL
fi
done
touch ${LAST_CHECK}
What the new code does is it checks against a check/touch file for its time stamp to get a list of newer log files to run on. When it finds log files that are newer than the touch file, it will process what is in the do loop. But regardless of whether this script runs, it touches the check/touch file.
How to tail a server log file and show only intended entries?
Dec 29th
Most of us working with JBoss or Tomcat server / container need to check log files for debugging information. What if we want the system to be able to filter out only the lines we want to see?
A typical log file generated by the server looks like this:
DEBUG [15:13:56] [prod.idms.service.impl.AuditLogServiceImpl.triggerEvent]= Checking against monitor: IDMS_USER_UPDATE_PWD
DEBUG [15:13:56] [prod.idms.service.impl.AuditLogServiceImpl.triggerEvent]= Checking against monitor: IDMS_USER_REGISTER_ADMIN
DEBUG [15:13:56] [prod.idms.service.impl.AuditLogServiceImpl.triggerEvent]= Checking against monitor: IDMS_USER_REGISTER
DEBUG [15:13:56] [prod.idms.service.impl.AuditLogServiceImpl.triggerEvent]= Checking against monitor: IDMS_USER_SUSPENDED
DEBUG [15:13:56] [prod.idms.service.impl.AuditLogServiceImpl.triggerEvent]= Checking against monitor: SYSTEM_BOOT_UP
DEBUG [15:13:56] [prod.idms.service.impl.AuditLogServiceImpl.triggerEvent]= Checking against monitor: SYSTEM_SHUTDOWN
INFO [15:13:56] [org.springframework.orm.hibernate3.AbstractSessionFactoryBean.destroy]= Closing Hibernate SessionFactory
INFO [15:14:02] [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService.unbindConnectionFactory]= Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=jdbc/ebarges' from JNDI name 'java:jdbc/ebarges'
INFO [15:14:02] [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService.unbindConnectionFactory]= Unbound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' from JNDI name 'java:JmsXA'
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: topic/testTopic
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: topic/securedTopic
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: topic/testDurableTopic
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: queue/testQueue
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: queue/A
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: queue/B
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: queue/C
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: queue/D
INFO [15:14:02] [org.jboss.mq.server.jmx.DestinationMBeanSupport.stopService]= Unbinding JNDI name: queue/ex
Lets say we want to only see on the fly generated log entries beginning with DEBUG?
$ tail -f ebarges.log | grep ^DEBUG
The results?
$ tail -f ebarges.log | grep ^DEBUG
DEBUG [15:20:34] [prod.ebarges.util.poller.DirectoryPoller.a]= [-- Monitoring "C:\c1\dev_project\ebarges\application_data\server\Incoming" --]
DEBUG [15:20:34] [prod.ebarges.util.poller.DirectoryPoller.a]= [-- Monitoring "C:\c1\dev_project\ebarges\application_data\server\Processed_Retry" --]
DEBUG [15:20:34] [prod.idms.service.impl.AuditLogServiceImpl.init]= AuditLogServiceImpl Registering: IDMS_USER_LOGIN
DEBUG [15:20:34] [prod.idms.service.impl.AuditLogServiceImpl.init]= AuditLogServiceImpl Registering: IDMS_USER_LOGOUT
DEBUG [15:20:34] [prod.idms.service.impl.AuditLogServiceImpl.init]= AuditLogServiceImpl Registering: IDMS_USER_LOGIN_FAILURE
DEBUG [15:20:34] [prod.idms.service.impl.AuditLogServiceImpl.init]= AuditLogServiceImpl Registering: IDMS_USER_UNAUTHORIZED_ACCESS
DEBUG [15:20:34] [prod.idms.service.impl.AuditLogServiceImpl.init]= AuditLogServiceImpl Registering: IDMS_USER_UPDATE_PWD
DEBUG [15:20:34] [prod.idms.service.impl.AuditLogServiceImpl.init]= AuditLogServiceImpl Registering: IDMS_USER_REGISTER_ADMIN
DEBUG [15:20:34] [prod.idms.service.impl.AuditLogServiceImpl.init]= AuditLogServiceImpl Registering: IDMS_USER_REGISTER
Installing Omeka CMS On Linux (CentOS)
Nov 17th
Omeka, although as weird as it sounds for a word that is hardly found in any legit dictionary, is an open source and free CMS that is quite simple to understand and capable of integrating into your development projects – as a CMS, to leave you to concentrate on more important aspects of your other software development. And although it is not as robust as Microsoft Sharepoint document management system in terms of its ease of integration as mentioned above, i personally find it interesting and quite relevant in the context of a project that i was recently involved in that i would recommend to you to take a look.
I begin and end by showing you around how Omeka would be installed under a GNU/Linux Kernel environment (i quite have to pretentiously or not put ‘Linux’ as we mostly know, as ‘GNU/Linux Kernel’ so as to respect all my acquaintance who place the distinction between Linux as a GNU/Linux kernel, and the ‘Linux’ we call any general Linux products. And of course i must agree it should be called GNU/Linux Kernel.)
Our target distribution appears to be CentOS which i have come to know and love, CentOS is a fork of RHEL as most would agree with, it is capable as a stable enterprise computing GNU/Linux kernel based server, and something that i would highly recommend for Omeka CMS to be installed and run on.
Because Omeka CMS runs on a HTTP server such as Apache HTTPD, we will step through installing Apache HTTPD in your operating system.
Prerequisite / Imposed Working Environment
1. You have installed in your server CentOS 5.5 or similar CentOS operating system version number, such as 4, 5.1, 5.2, 5.3 or its RHEL equivalent.
2. You have access to the Internet as we need to download certain packages and installation files related to the setup.
3. You have root access or may execute sudoer commands in the server.
4. You have installed in your server MySql DBMS server.
5. You have the necessary PHP installation in your server.
Required Installations (Which we will walk through in this guide)
1. Apache HTTPD
Installing Apache HTTPD Web Server
1. By using the yellowdog updater modified package manager of CentOS, installing Apache HTTPD is as simple as issuing a ‘yum install httpd’ command.
$yum install httpd
Upon issuing the ‘yum install’ command, the package manager searches for the package to be installed, outputs some verbose information and asks for decision to continue to download the installation files and any dependencies that need to be updated and installed along with the HTTPD.
Verbose information being output, the yellowdog (yummy) updater modified package manager searches for dependencies and suggests to you what it intends to do with you to choose between yes or no. Choose ‘Y’ for yes to proceed.
Yum install completed.
2. (Optional) Configure Apache HTTPD to start automatically following a start up or restart.
You may configure Apache HTTPD to start automatically after a server restart or a cold start up. It is generally a good idea if you have many servers under your charge, it is not just for the particular reason of easing your workload or responsibilities. Circumstances following a warn restart usually are perplexing and there may be many other things you want to skip through to get to the main agenda, investigation or patching for example.
$chkconfig –level 2345 httpd on
The chkconfig command allows you to specify the state of the runtime (1 to 6) of any particular software. The above command will ensure Apache HTTPD is started automatically.
You may double configure this by checking through chkconfig –list httpd.
$chkconfig –list httpd
Apache HTTPD in set to ‘on’.
3. Start Apache HTTPD web server.
$service httpd start
(There are some irrelevant output from the server as seen in the screenshot, this is due to some configurations i did not set on my server, i am using a virtual machine for this example, you shouldn’t be having this)
Installing Omeka
At this point we have installed in our server Apache HTTPD which the only additional setup we need to perform before installing Omeka.
And at this point, Omeka is not installable via $yum install omeka, which requires us to manually download, unzip and manage the packaging, directory ourselves.
Omeka is available for download from here although it may quite possibly change from time to time. And in this case nevertheless, Google is your best mate.
1. Download omeka from your server.
$wget http://omeka.org/files/omeka-1.2.1.zip
Download completed
Doing a quick listing ($ls -l) shows the installation file that is downloaded to the server.
2. Unzip the installation package into /var/www/html/
The unzip command unzips the zip package to the current directory by default, you may have noticed that i have sent in a few parameters and pipes. Let me explain these options.
The unzip command by default verbosely outputs all the directory and files that it is unzipping to, to the screen, and what you get are things you not necessarily need to ‘scan’ through. the pipe -q quietly unzips.
Because the default behavior of the unzip command unpacks all files into the same directory of the zip file and our intention is to place it into the /var/www/html where Apache HTTPD is configured by default to serve from, we pipe in -d /var/www/html.
3. Go to your unzip directory
We will go to our unzip directory, /var/www/html using the “cd” command and “ls -l” to list.
We will also change the verbose name of the default installation directory name from “omeka-1.2.1″ to simply “omeka” using the “mv” move command.
4. Change ownership of the Omeka installation from “root” to “apache”
For Apache HTTPD to effectively manage Omeka, change the ownership of the entire Omeka installation to “apache” from whatever it was.
Notice that i have in my screenshot showed the effects of the chown command on the targeted directory before and after.
The flag -R specifies that the command will traverse every folder under the folder specified and apply the same owner and group (apache:apache), essentially, everything.
5. Configure the DBMS connection settings through db.ini
The db.ini find can be found in the first level directory of omeka which currently resides on /var/www/html/omeka/. Using your favorite text editor, edit the text file.
I would be using the text editor vim for this guide.
By default, db.ini shows default values in which we must change out.
host: This can be localhost, the ip equivalent of localhost (as specified by your /etc/hosts or by default 127.0.0.1), the domain name or ip of where the DBMS resides on.
username: Self explanatory.
password: Self-explanatory.
name: The name of the database, this is the name you supplied when you created your database for use with omeka (create database abc; for example).
prefix: The use of prefix is commonly found in shared hosting facilities where one has limitations over the number of mysql database server or instances for the package paid for. It is perfectly fine for you to leave this empty otherwise, as it would not serve any use other than the above mentioned multiple databases enabling “loophole”.
6. Finish off installation through web browser.
Once we are past this stage, we may finish off the rest of the installation and customization of Omeka through the web browser.
Complete!
Special thanks to Eugene for the comprehensive explanation and practices on chkconfig!
Unix Command Find And Delete The Efficient Way With xargs
Oct 24th
$find . -name “*.h” -print0 | xargs -0 rm
Conventionally to delete a file in linux you would issue the rm command, rm for ReMove.
This command below is for removing one file, named abc.c
$rm abc.c
To remove multiple files, we may pipe in the known list of files into the arguments of rm.
$rm abc.c dfr.c snh.c
To remove all files in the directory you may issue a wild card in many forms.
$rm *.c
$rm *.*
$rm *
What if you wish to find and delete, such that you would like to use the more complex features of Unix find command to locate files and then deleting them in a single OR minimal remove command.
When issuing this command, the executor attempts to find all the files that qualify under the specified directory and then pipes into xargs which pipes as many file directory into rm as possible for single execution.
Introducing the Find + Remove command
$find -name “< qualifier >” -print0 | xargs -0 rm
Sample Usage.
$find /home/ken -name “*.cpp” -print | xargs -0 rm(Mistake / Erroneous)$find /home/ken -name “*.cpp” -print0 | xargs -0 rm
$find /home/ken -name “touch.*” -print | xargs -0 rm(Mistake / Erroneous)$find /home/ken -name “touch.*” -print0 | xargs -0 rm
But Why Use This Instead of Say -exec rm ‘{}’ \;
As mentioned, the advantage of xargs is it passes and packs as many arguments into rm as possible. Using -exec executes a remove command for each file found, which is less efficient.
Thanks to Eugene for teaching me this! Cheers.
Corrected with comments from ole.tange.dk (With thanks!)
Last Edited 2010-10-25 10:05:00AM
