Intstall "Ruby on Rails" on Ubuntu 9.04 and Dont Repeat Yourself Anymore

My new position during initiating days of the new Iranian's year just started by a weird project that outsourced by an over seas company to an Iranian one.

I have cooperated in some software projects with different technologies such as Java, .Net and Delphi. Indeed, the latest one is a Ruby on Rails web based project that I have selected by the boss as the technology specialist and the senior developer to find out the way of using this unfamiliar technology. The project requirements and specifications ride us toward cutting edge technologies. For the first day searching and surfing through technical pages have filled my working time.

I remember some months ago, a friend of mine tried to hire some Ruby on Rails developers in Tehran. He didn't succeed. This is why I feel so lucky. Ruby on Rails is designed based on the “Convention over Configuration” idea, and the rapid development principle of “Don't Repeat Yourself(DRY)”. I have found its agile motivations so practical. I as a web developer have wast a vast of time to implement repeated and simple parts of the applications such as database connectivity and configurations. Ruby on Rails reduces this times by emphasizing CoC and DRY concepts.

Ruby on Rails is often installed using RubyGems, a package manager which is included with Ruby. Many Linux flavors also support installation of Rails and its dependencies through their native package management system.
I tried to deploy it on Ubuntu 9.04. It wasn't easy and straight forward in my case. I tried a couple of times.

I found that I have to install ruby-full and ruby-elips packages first. Next, its time to install Ruby and Rails. Leave RubyGems for the latest installation. After all you need an IDE. RadRails is an Eclipse based Ruby on Rails IDE that facilitating the developments using its nice and fast tools and services.

Here are all commands in detail:
First remove all installed Ruby, Rails, Gems and everything you will install in next steps. Any remain parts may make some troubles for this process.

Make sure your repository is updated:
sudo apt-get update

Install rails by below command:
sudo apt-get install ruby ri rdoc irb libopenssl-ruby ruby-dev

Download RubyGem.

Extract it and then install Gem by following commands:
tar xvzf rubygems-1.3.5.tgz
cd rubygems-1.3.5
sudo ruby setup.rb

You don't need to the downloaded file and extracted folder any more. So remove it:
rm -r rubygems-1.3.5 rubygems-1.3.5.tgz

Now, create a set of simlinks:
sudo ln -s /usr/bin/gem1.8 /usr/local/bin/gem
sudo ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby
sudo ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc
sudo ln -s /usr/bin/ri1.8 /usr/local/bin/ri
sudo ln -s /usr/bin/irb1.8 /usr/local/bin/irb

Make sure ruby-dev installed:
sudo apt-get install ruby-dev

Next, install the Rails:
sudo gem install rails

You need Mongrel as server:
sudo gem install mongrel passenger capistrano

Setup databases (sqlite3 and MySQL):
sudo apt-get install sqlite3 swig libsqlite3-ruby libsqlite3-dev
sudo gem install sqlite3-ruby
sudo apt-get install mysql-client libmysqlclient15-dev
sudo gem install mysql

Test what have you done:
rails test_app

Test the application with MySQL support:
rails test-app -d mysql

Run the application:
cd test_app
script/server

Parsing Huge Text Files Using Java and JSapar

Last week a friend of mine and I decided to parse a huge size text file that consists some reports of legacy devices. After few times trying we got that, opening and parsing huge text files in Java is very time and resource consuming. We started with a 35MB log file. We have never worked with such a huge size text files. So we tried to find the relevant solution. Indeed, Java is not the best solution for this kind of problems. I believe Python or Perl could perform this requirement by a higher performance. However regards to later developments and project requirements we decided to use Java. After some searching through web we found a brilliant tool. Tigris.org has some valueable open source projects. JSapar is one of them. JSapar is a Java library providing a schema based parser/producer of CSV (Comma Separated Values) and flat files. The goal of this project is to create a java library that contains a parser of flat files and csv files.
The file imports in to an object oriented model that we called it telegrams. The parser produces a Document class, representing the content of the file, or you can choose to receive events for each line that has been successfully parsed. Tigris claims that JSapar can handle huge files without loading everything into memory.
The library is simple to use and possible to extend. Our log file consists thousands of lines just the same as below sample line:

948853 : 47 E6 18 FF 04 CD 0B 1D B1 C1 D1 1E ;

This is a telegram. First part is row number (948853) and next bytes contains a message. This two part are separated by a ":". At first sight it seems it is a straight forward procedure, however, it is not as much easy as it looks. Millions of these lines makes a real slow running and unstable application if you use standard java scaner or parsers. First we defined a schema for csv files:

<?xml version="1.0" encoding="UTF-8"?>

<schema xmlns="http://jsapar.tigris.org/JSaParSchema/1.0">

<csvschema lineseparator="\n">

<line occurs="*" linetype="Telegram" cellseparator=":">

<cell name="Row No" />

<cell name="Body"/>

</line>

</csvschema>

</schema>

Then we just used a simple java code to read a 40MB text file into memory in less than 10 seconds.

public final void loadTelegrams() throws SchemaException, IOException, JSaParException {
Document telegrams;
Reader schemaReader = new FileReader("schema/schema.xml");
Xml2SchemaBuilder xmlBuilder = new Xml2SchemaBuilder();
Reader fileReader = new FileReader("repo/dat.txt");
Parser parser = new Parser(xmlBuilder.build(schemaReader));
telegrams = parser.build(fileReader);
fileReader.close();
}

Using below command we moves whole of the file cell by cell so quickly.

telegrams.getLine(i).getCell(j).getStringValue();

OFBiz, The Apache Open for Business

I am writing this to introduce the next generation of web based business applications that I am developing. As you know I used to develop a small J2EE based application that I called it Easy Tracking. Easy Tracking consists inventory, orders and sales applications. Also it has some small features and a rich Ajax based multi-language client. This is more than 4 years that my customers use it as a rental software and data hosting service on my servers. I have gained a lot of valuable experience in this kind of business service providing among the years.
Now, different requests of individual customers makes me interested to use an open source professional business solution. During last months I reviewed a vast of ERP solutions and finely I have decided to use OFBiz. OFBiz is an Apache product; moreover, I reviewed its architecture and technologies and found them all matured and practical. Actually OFBiz is an open source framework to help developers in creating high quality business solutions. Development and customization both are available using straight forward procedures. However, OFBiz needs a mass of modification and development to be adapted with most of Iranian business companies requirements. In addition, it doesn't have a Farsi face yet.
After a while I will need a patient rich customer to let me implement his business requirements by the solution and a holding software company who needs to have a localized, reliable and ready to use product.

Distributed Application Debugging Tips

Every distributed application works based on a collaborative environment that consists of services and method calls. Moreover, in a distributed environment almost services are hosted on different platforms and far machines that talks through network. Distribution makes some difficulties in the debugging of this kind of applications.

During the development of a J2EE based core banking software, I as a member of the team have had many struggled challenges for debugging. The solution uses some far, distributed components through network that work together using message queuing following SOA disciplines.

DB2 Database, IBM MQ, IBM Websphere and Swing reach client are talking together using a vast of message sending and receiving. Also architecture uses Mule ESB, Spring, JMS and a vast of configuration files. We have found and fixed most of bugs using below simple techniques.

Use a Map. Just draw a symbolic map of the environment that causes the problem. Highlight servers, clients, firewalls, routers, with their specific IP and ports of each one that used in the scenario. May be an IP, a port or a firewall policy has been changed and the problem has arisen. Using Ping and Telnet be sure that service listeners are available at least.
If the service was working and a problem has arisen eventually; then, configuration changing is the most error prone factor.

Interpret Log Files. Review log files and exception messages carefully. Almost every raised exception message points to the problem exactly. In the other hand, logging is not a centralized activity in a layered distributed application. Indeed, each layer or component may logs its exceptions separately. So check them all. Check log files just like a detector and use your imagination to guess the problem.

Go Deeper. Adjust logging level to the proper value to let logger catch more detail messages.

Set Break Points to Watch. Set some break points to watch out what is happening during run time. Be sure messages send and receive by end points correctly. Then, step toward inner layers to find out what happened.

Use Replacements. Replace the component you suspected in with the correct one to find out if they are working probably. For example, you have another available application server, message queue or database use them regard the situation.

Check Configuration Files. Be sure the build routines has done their tasks perfectly. Maven and Ant write values of variables in the compiled version of configuration files. Such as context.xml, web.xml and other xml files. Lack of privilege or wrong configuration may prevent build process to finish its task completely. So check the compiled and built version of configuration files.

Be Patient. In a distributed system, method call doesn't perform as much fast as the standalone application. So check if timeout values are enough or not. Sometimes increasing timeout value is a key to solve the problem.

Do it Faster. Faster compiling and running on lighter machines is the an effective approach to test more situations rapidly. So use lighter application servers and databases. For example, use Tomcat over Websphere; Also, use MySQL rather than DB2 during development.

Routing multiple domains to an application using URL rewrite filter in JSP contatiners and JEE application servers

My client has assigned two separated domain address to a certain IP address. My client was going to map two domain addresses to an application which hosted on JBoss 4.2.2.
Each domain address should route requests to some certain pages of the application. I had to assign firstdomain.com and seconnddomain.com to the a certain application that I call it "app" here. My task was clear; indeed, I had to implement below mapping:

• http://firstdomain.com .... http://1.2.3.4/app/first.html
• http://seconddomain.com .... http://1.2.3.4/app/repo/second.html

As you can see, IP and application name are the common parts and inner paths goes different. I used UrlRewriteFilter that simulates Apache rewrite URL method in JSP containers and JEE app servers. It is just enough to follow the instruction. UrlRewriteFilter has a vast of samples but I have not found the proper rule that maps different url on a certain IP address. So, after some tries and errors; finally, below rule worked for me:

<rule>

<name>Changing Domain</name>

<condition name="host" operator="equal">firstdomain.com</condition>

<from>^/(.*)</from>

<to type="permanent-redirect">http://1.2.3.4/app/repo/second.html$1</to>

</rule>

This is just enough to add above rule into the urlrewrite.xml.

first.html is the default page. So you just need to handle the request which comes to the seconddomain.com. I have got my routing goals on Tomcat 6.0 and JBoss 4.2.2 with this simple URL re-writing.

Why I use Google Docs

Following extending an office automation, I worked on Apache Jackrabbit content repository some years ago. After it, I was looking for a free personal content repository to keep my documents, sample codes and commands over HTTP. Something just like a personal wiki, personal blog or a repository.
I preferred to use a free one that supported by the web giants such as Google and Yahoo. Finally, I have found it. My friends in Facebook suggested me to try Google Documents two weeks ago. Google Docs is exactly the tool I was looking for. Google means this is improving faster and wiser than others. Moreover, Google Documents is the most interesting tool that I found in the Google shelf. It works very fast; moreover, it is pretty much. Google Docs enables me to keep my valuable documents in a safety box on WWW. I shared some documents with the trusted friends of mine. Also, a friend and I are using Google Docs in an amazing way to keep synchronize together around an educational program. In addition, Google Docs, keeps all revisions on modifications. In fact, you can track the document growing life cycle. In my opinion every body should have his or her Google Documents account because of below reasons:

• Your knowledge is always accessible.
• You don't need to care about hard disk failures and viruses attack any more.
• You can edit your documents as well as possible just using a browser.
  
• Google Docs is a repository so, all changes are traceable.
• Google Docs Sharing mechanism works pretty secured.
• I don't care about currently 1024 MB capacity limitation. Google is growing always.
• Good feeling of extending my memory using Google Docs features is another special kind of experience that I gain by using Google Documents. You will never loose it if you keep it by Google Documents.

Good points for putting breaks across Mule ESB classes

This is the second year that I work as a J2EE developer during development of a modern J2EE based core banking project. The architecture uses Spring and Mule ESB as its major components widely. Mule ESB (Enterprise Service Bus) allows us to connect the applications together as much simple as possible. The architecture uses Mule ESB as integration platform and service container. Mule ESB provides some easy to use methods for talking through JMS, HTTP, FTP and many more transfer protocols. Moreover, Mule ESB implemented a set of relevant mechanisms for message routing, message transformation and transaction management. However, debugging the started services which use Mule ESB functionalities is not an easy activity. Because there is a lot of layered listeners and complex network devices/services which make problems bolder. The framework we used is a well design art work of my friend, Ara Abrahamian and it works like a Mercedes Benz in most of areas. But, the modules added by developers almost have some problems. I noticed most of problems that we raise, has the origin in the wrong configuration files such as muleesb-config.xml and Spring's context.xml. Also, below classes could be a good place to start debugging:

• JmsMessageDispatcher.java
• JmsMessageConsumerImpl.java

JmsMessageDispatcher is the the most outer place just before sending message. Also, JmsMessageConsumerImpl is the first JMS message receiver.

In my experience debugging this two classes is an easy way to find what happened to the code or underlying services malfunctions.

JBoss container managed basic authentication

These days I am working on an ACS (Automatic Configuration System) which runs on a JBoss app server. So, yesterday I configured the server for cotnainer-managed authentication. Here is a straight forward tutorial to demonstrate how to secure your applications which hosted in JBoss by BASIC authentication methods. Container-managed authentication methods control how a user's credentials are verified when a web app's protected resource is accessed. This is a straight forward tutorial that shows how to configure JBoss application server to protect resources by a role based simple mechanism. By receiving any request to the protected resources JBoss asks web browser for a username and password. With this authentication method, all passwords are sent across the network in base64-encoded text. This could be fine to review the commented parts of JBoss configurations to find some more facilities If you need.

First step, web.xml modifications
Find the web.xml in the WEB-INF folder of your deployed ear application. This could be find in somewhere like this depends on the OS you use. Add below lines at the bottom of web.xml just before tag.

<security-constraint>
<web-resource-collection>
<web-resource-name>ossmanager</web-resource-name>
<url-pattern>/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>ossmanager</role-name>
</auth-constraint>
<user-data-constraint>
<transport-guarantee>NONE</transport-guarantee>
</user-data-constraint>
</security-constraint>

<login-config>
<auth-method>BASIC</auth-method>
<realm-name>OSS Security</realm-name>
</login-config>

<security-role>
<description>ossmanager is the role required to access under oss</description>
<role-name>ossmanager</role-name>
</security-role>

<security-role>
<description>cpeowner is the role required to access under cpeowner</description>
<role-name>cpeowner</role-name>
</security-role>

Second step, joss-web.xml modifications
Right besides web.xml in the WEB-INF folder you can find jboss-web.xml to add below lines in. Note, datak-oss is the context of the application and acs-security-policy is the optional name that you call your security policy using it.

<jboss-web>
<context-root>datak-oss</context-root>
<security-domain>java:/jaas/acs-security-policy</security-domain>
</jboss-web>

Third step, login-config.xml modifications
Find login-config.xml in server/default/conf folder and add below lines at the proper place.

<application-policy name ="acs-security-policy" >
<authentication>
<login-module code="org.jboss.security.auth.spi.UsersRolesLoginModule" flag="required">
<module-option name="usersProperties">acs-users.properties</module-option>
<module-option name="rolesProperties">acs-roles.properties</module-option>
</login-module>
</authentication>
</application-policy>

Create acs-users.properties and acs-roles.properties text files into the WEB-INF/classes/ the same as below:

acs-users.properties:
amir=takeiteasy
kamran=mamooshi

acs-roles.properties:
amir=ossmanager,cpeowner
kamran=cpeowner

Exclude a URL by adding a blank security-constraint
I added this part two days later. Because the project manager asked me to exclude a certain URL (/cpe) from the secured zone. I have performed it just adding a new without role constraint. Below code added an excluded path to the secured zone, which every anonymous user can access it. Add below security-constraint to the web.xml the same as security-constraint that you added before.

<security-constraint>
<web-resource-collection>
<web-resource-name>CPE</web-resource-name>
<url-pattern>/cpe/*</url-pattern>
</web-resource-collection>
</security-constraint>

Make programming a bit easier in Linux

Restarting services or killing them are the must repeated activities that I have to do during current project. Running maven commands and deleting '.svn' folders are so easy but those take time too. By the other hand all of them should accomplish in the operating system shell. Sometimes more than ten times per a day I have ran this commands. Moreover, everyone knew that it could be much easier to develop simple Linux bash scripts or define Aliases which contains more usage commands. Then, I developed below scripts and define some aliases to make it easier.

killj.sh is a bash script which kills my Java processes that consists 'javaagent' keyword. The application needs 'javaagent' as VM parameter; so, I am looking for it to kill the right process.

#!/bin/bash
for i in `ps -ef | grep javaagent | cut -c10-14`
do
/bin/kill $i
echo " $i has killed."
done
if [ $(ps ax | grep -c javaagent) -gt 1 ] ; then
echo "NOTE: There are more alive java process yet!!!"
else
echo "DONE"
fi

rmsvn.sh removes .svn folders recursively.

#!/bin/bash

echo

echo "Warning!!!"

echo "CHECK BELOW PATH. This will remove all .svn folders recursively!!!"

pwd

read -p "Are you sure (yes/no)? "

[ "$REPLY" == "yes" ] && rm -rf `find . -type d -name .svn` && echo "all .svn folders have been removed!!!"

Also I added below lines at the end of .bashrc file:

alias lt='ls -alt | head -20'

alias ld="ls -al -d * | egrep '^d'"

alias c="mvn clean"

alias ci="mvn -Pdev -DskipTests clean install"

alias i="mvn -Pdev -DskipTests install"

alias jkill="/bin/killj.sh"

alias rmsvn="/bin/rmsvn.sh"

alias go="cd /home/amir/projects/company/jcb2"

alias dpiproxy="sudo cp -f /etc/tor/torrc.dpi /etc/tor/torrc"

alias noproxy="sudo cp -f /etc/tor/torrc.noproxy /etc/tor/torrc"

alias tor="sudo /etc/init.d/tor restart"

Now, compiling, cleaning and killing processes are much easier...

Port mapping in Linux in Ubuntu and CentOS

If you used to setup web servers under windows easily, you will find Linux doesn't show itself bared on the network. Indeed, Linux has its own secured solutions to let your web servers listen to the 80 port. In fact, you can not use the ports under 1024 for the individual users. Because they belongs to the super user. Moreover, its recommended to run web servers with a limited user. So, the best way to run HTTP listeners on 80 port is to map upper than 1024 ports 80 port.
Last week I worked with a JBoss application server which has ran on 8080 port. In addition, the client needed to run it on 80 port, without any modification on JBoss settings. I make it so easily by running 3 iptable commands which I found in some tutorials on the web.

sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

sudo iptables -I INPUT -p tcp --dport 8080 -j ACCEPT

sudo iptables -t nat -A OUTPUT -o lo -p tcp --dport 80 -j REDIRECT --to-port 8080

NOTE: if you dont put -o lo option for the last command, all outgoing requestes will rout to the 8080 port on the localhost.

Finally you should save new rules into disk. below command persists changes.

sudo iptables-save > iptable.rules

Moreover you need to make sure rules will loaded in the next start up. So, add below line into /etc/network/interfaces after iface command:

pre-up iptables-restore < /etc/iptable.rules

If you are using CentOS then it is enough to run below command to persist iptables rules:

/sbin/service iptables save

Restart the network connection and enjoy it.