Saturday, August 21, 2010

Ruby on Rails on CentOS 5.5 with REE/Phusion Passenger

Install (minimal) gcc toolchain


$ sudo yum install autoconf automake binutils gcc gcc-c++

Install Ruby Enterprise Edition (REE)


Download Ruby Enterprise Edition


Extract & run installer


tar xzvf ruby-enterprise-X.X.X.tar.gz
./ruby-enterprise-X.X.X/installer

Welcome to the Ruby Enterprise Edition installer
This installer will help you install Ruby Enterprise Edition 1.8.7-2010.02.
Don't worry, none of your system files will be touched if you don't want them
to, so there is no risk that things will screw up.

You can expect this from the installation process:

1. Ruby Enterprise Edition will be compiled and optimized for speed for this
system.
2. Ruby on Rails will be installed for Ruby Enterprise Edition.
3. You will learn how to tell Phusion Passenger to use Ruby Enterprise
Edition instead of regular Ruby.

Press Enter to continue, or Ctrl-C to abort.

Checking for required software...

* C compiler... found at /usr/bin/gcc
* C++ compiler... found at /usr/bin/g++
* The 'make' tool... found at /usr/bin/make
* The 'patch' tool... found at /usr/bin/patch
* Zlib development headers... not found
* OpenSSL development headers... not found
* GNU Readline development headers... not found

Some required software is not installed.
But don't worry, this installer will tell you how to install them.
Press Enter to continue, or Ctrl-C to abort.

--------------------------------------------
Installation instructions for required software

* To install Zlib development headers:
Please run yum install zlib-devel as root.

* To install OpenSSL development headers:
Please run yum install openssl-devel as root.

* To install GNU Readline development headers:
Please run yum install readline-devel as root.

REE installer gives me a nice list of dependencies I am missing. So lets install them


$ sudo yum install zlib-devel openssl-devel readline-devel 


Checking for required software...

* C compiler... found at /usr/bin/gcc
* C++ compiler... found at /usr/bin/g++
* The 'make' tool... found at /usr/bin/make
* The 'patch' tool... found at /usr/bin/patch
* Zlib development headers... found
* OpenSSL development headers... found
* GNU Readline development headers... found
--------------------------------------------
Target directory
Where would you like to install Ruby Enterprise Edition to?
(All Ruby Enterprise Edition files will be put inside that directory.)
[/opt/ruby-enterprise-1.8.7-2010.02] :

I am fine with this location, so I hit enter


Warning: some libraries could not be installed
The following gems could not be installed, probably because of an Internet
connection error:

* mysql
* sqlite3-ruby
* pg

These gems are not required, i.e. Ruby Enterprise Edition will work fine without them. But most people use Ruby Enterprise Edition in combination with Phusion Passenger and Ruby on Rails, which do require one or more of the aforementioned gems, so you may want to install them later.

To install the aforementioned gems, please use the following commands:
* /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby /opt/ruby-enterprise-1.8.7-2010.02/bin/gem install mysql
* /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby /opt/ruby-enterprise-1.8.7-2010.02/bin/gem install sqlite3-ruby
* /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby /opt/ruby-enterprise-1.8.7-2010.02/bin/gem install pg

I don’t need Postgress, but I do need the other two


$ sudo yum install mysql-server mysql mysql-devel
$ sudo yum install sqlite-devel

Configure mysqld to run at system startup


$ sudo /sbin/chkconfig --list | grep mysqld
mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off
$ sudo /sbin/chkconfig --level 35 mysqld on

Now install the necessary gems


$ sudo /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby /opt/ruby-enterprise-1.8.7-2010.02/bin/gem install mysql
Building native extensions. This could take a while...
Successfully installed mysql-2.8.1
1 gem installed

And sqlite3-ruby


$ sudo /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby /opt/ruby-enterprise-1.8.7-2010.02/bin/gem install sqlite3-ruby
Building native extensions. This could take a while...
ERROR: Error installing sqlite3-ruby:
ERROR: Failed to build gem native extension.

/opt/ruby-enterprise-1.8.7-2010.02/bin/ruby extconf.rb
checking for sqlite3.h... yes
checking for sqlite3_libversion_number() in -lsqlite3... yes
checking for rb_proc_arity()... no
checking for sqlite3_initialize()... no
sqlite3-ruby only supports sqlite3 versions 3.6.16+, please upgrade!

Looks like sqlite3-ruby needs a higher version of sqlite. So I built sqlite from source


$ wget http://www.sqlite.org/sqlite-amalgamation-3.7.0.1.tar.gz
$ tar xvzf sqlite-amalgamation-3.7.0.1.tar.gz
$ cd sqlite-3.7.0.1
$ ./configure --prefix=/opt/local/sqlite-3.7.0.1
$ make
$ sudo make install

And now re-install sqlite3-ruby gem


$ sudo /opt/ruby-enterprise-1.8.7-2010.02/bin/gem install sqlite3-ruby -- --with-sqlite3-dir=/opt/local/sqlite-3.7.0.1 
Building native extensions. This could take a while...
Successfully installed sqlite3-ruby-1.3.1
1 gem installed

Install Apache


$ sudo yum install httpd-devel

Configure httpd to start at system startup


$ sudo /sbin/chkconfig --list | grep httpd
httpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
$ sudo /sbin/chkconfig --level 35 httpd on

Start httpd


$ sudo /sbin/service httpd start
Starting httpd: [ OK ]

Open port 80 on the firewall


$ vi /etc/sysconfig/iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Notice, I added the line -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 80 -j ACCEPT

Restart firewall for changes to take effect


$ sudo /sbin/service iptables restart
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
Applying iptables firewall rules: [ OK ]
Loading additional iptables modules: ip_conntrack_netbios_n[ OK ]

Install Phusion Passenger (modrails)

See Phusion Passenger for further details


$ sudo gem install passenger
[sudo] password for amitava:
Building native extensions. This could take a while...
Successfully installed passenger-2.2.15
1 gem installed
Installing ri documentation for passenger-2.2.15...
Installing RDoc documentation for passenger-2.2.15...
[amitava@gestpedia-dev test]$ sudo passenger-install-apache2-module
The Apache 2 module was successfully installed.
Please edit your Apache configuration file, and add these lines:
LoadModule passenger_module /opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/passenger-2.2.15/ext/apache2/mod_passenger.so
PassengerRoot /opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/passenger-2.2.15
PassengerRuby /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby

After you restart Apache, you are ready to deploy any number of Ruby on Rails
applications on Apache, without any further Ruby on Rails-specific
configuration!

Add mod_passenger configuration


$ sudo vi /etc/httpd/conf.d/modrails.conf

LoadModule passenger_module /opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/passenger-2.2.15/ext/apache2/mod_passenger.so
PassengerRoot /opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/passenger-2.2.15
PassengerRuby /opt/ruby-enterprise-1.8.7-2010.02/bin/ruby

$ sudo /sbin/service httpd restart
Stopping httpd: [ OK ]
Starting httpd: httpd: Syntax error on line 210 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.d/modrails.conf: Cannot load /opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/passenger-2.2.15/ext/apache2/mod_passenger.so into server: /opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/passenger-2.2.15/ext/apache2/mod_passenger.so: failed to map segment from shared object: Permission denied
[FAILED]

Oops! I hit trusted google and get the following links

Goolge group thread

modrails User Guide

So SELinux is the culprit. Setup the SELinux context


$ sudo chcon -R -h -t httpd_sys_content_t /opt/ruby-enterprise-1.8.7-2010.02
$ sudo /sbin/service httpd restart
Stopping httpd: [FAILED]
Starting httpd: [ OK ]

Deploy the rails application

See User Guide

Create a directory for your webapps, and move (or copy) your app to that location


$ sudo mkdir /var/www/webapps
$ sudo mv coolapp /var/www/webapps/
$ sudo ln -s /var/www/webapps/coolapp/public /var/www/html/coolapp
$ sudo vi /etc/httpd/conf.d/coolapp.conf
<VirtualHost *:80>
ServerName www.coolapp.com
RailsBaseURI /coolapp
<Directory /var/www/html/coolapp>
Options -MultiViews
</Directory>
</VirtualHost>

$ sudo /sbin/service httpd restart

$ sudo tail -f /var/log/httpd/error_log
[Fri Aug 20 15:35:45 2010] [notice] caught SIGTERM, shutting down
[Fri Aug 20 15:35:45 2010] [notice] SELinux policy enabled; httpd running as context user_u:system_r:httpd_t:s0
[Fri Aug 20 15:35:45 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Aug 20 15:35:45 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.20709/.guard: Permission denied (13)
[Fri Aug 20 15:35:45 2010] [notice] Digest: generating secret for digest authentication ...
[Fri Aug 20 15:35:45 2010] [notice] Digest: done
[Fri Aug 20 15:35:45 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.20711/.guard: Permission denied (13)

StackOverflow

This is still an SELinux issue. I found a working selinux policy here
I am copying it here for easy reference


$ vi passenger.te
module passenger 1.0;
require {
type httpd_tmp_t;
type devpts_t;
type httpd_sys_script_t;
type security_t;
type httpd_t;
type unconfined_t;
type selinux_config_t;
type hi_reserved_port_t;
type httpd_sys_content_t;
type var_t;
type cert_t;
class file { getattr read create append };
class process { siginh signal noatsecure rlimitinh };
class unix_stream_socket { read write shutdown };
class chr_file { read write };
class capability { setuid dac_override chown fsetid setgid fowner };
class fifo_file { setattr create getattr unlink };
class sock_file { write getattr setattr create unlink };
class lnk_file { read getattr };
class udp_socket name_bind;
class dir { write read search add_name };
}

#============= httpd_sys_script_t ==============
allow httpd_sys_script_t cert_t:dir search;
allow httpd_sys_script_t cert_t:file { read getattr };
allow httpd_sys_script_t cert_t:lnk_file read;
allow httpd_sys_script_t devpts_t:chr_file { read write };
allow httpd_sys_script_t httpd_sys_content_t:fifo_file setattr;
allow httpd_sys_script_t httpd_sys_content_t:sock_file { create unlink setattr };
allow httpd_sys_script_t httpd_t:unix_stream_socket { read write };
allow httpd_sys_script_t httpd_tmp_t:fifo_file setattr;
allow httpd_sys_script_t httpd_tmp_t:sock_file { write create unlink setattr };
allow httpd_sys_script_t self:capability { setuid chown fsetid setgid fowner dac_override };
allow httpd_sys_script_t unconfined_t:process signal;
allow httpd_sys_script_t var_t:dir { write read add_name };
allow httpd_sys_script_t var_t:file { read getattr create append };
#============= httpd_t ==============
allow httpd_t hi_reserved_port_t:udp_socket name_bind;
allow httpd_t httpd_sys_content_t:fifo_file { create unlink getattr setattr };
allow httpd_t httpd_sys_content_t:sock_file { getattr unlink setattr };
allow httpd_t httpd_sys_script_t:process { siginh rlimitinh noatsecure };
allow httpd_t httpd_sys_script_t:unix_stream_socket { read write shutdown };
allow httpd_t httpd_tmp_t:fifo_file { create unlink getattr setattr };
allow httpd_t httpd_tmp_t:sock_file { getattr unlink setattr };
allow httpd_t security_t:dir search;
allow httpd_t self:capability { fowner fsetid };
allow httpd_t selinux_config_t:dir search;
allow httpd_t var_t:file { read getattr };
allow httpd_t var_t:lnk_file { read getattr };

$ checkmodule -M -m -o passenger.mod passenger.te
$ semodule_package -o passenger.pp -m passenger.mod
$ sudo /usr/sbin/semodule -i passenger.pp

$ sudo /sbin/service httpd restart

Thursday, August 19, 2010

Scala wrappers for java collection


import scala.collection.JavaConversions._

System.getProperties.foreach(println)


Please note that the java method System.getProperties() returns java.util.Properties. There is no foreach defined in that class. Behind the scene, scala.collection.JavaConversions defines implicit conversions that converts java collections to scala collections.

Friday, August 6, 2010

Python titbits

I was generating sql data load scripts from excel documents using python. Here’s the code snippet
import csv

with open('cases.csv', 'rU') as f:
# r = csv.DictReader(f, dialect=csv.excel)
r = csv.reader(f, dialect=csv.excel)
cases = [row[:-3] for row in r]

Few interesting aspects in the code above

  1. I am using python’s with statement to automatically close the csv file after use.

  2. I am opening the file in ‘rU’ mode - i.e. readonly and Universal mode. ‘U’ is critical for csv module to parse excel generated csv’s. Here’s the gory details

  3. Use csv.DictReader if you want data in a dictionary keyed by the column names. DictReader exposes a
    convenient property ‘fieldnames’ that has the column names if your csv has a header row.


  4. Notice the use of python’s list comprehension to populate the cases array. Also, I am chopping
    off the last 3 columns (which was empty in the data file) using python’s slice operator.



For better clarity, I changed the code to
import csv
import collections

Case = collections.namedtuple('Case',"program, title, meeting_time, meeting_date")
with open('foo.csv', 'rU') as f:
# r = csv.DictReader(f, dialect=csv.excel)
r = csv.reader(f, dialect=csv.excel)
cases = [Case(*row[:-3]) for row in r]

namedtuple is similar to ruby’s Struct class.
This is nicely encapsulated now. The rest of the code need not worry about which column
the data was in the original data file. Also note the use * in Case(*row[:-3]). I am using * to pass the parsed array as arguments to Case. In clojure, you would use apply

Now consumers of cases can use it like so
qry = '''INSERT INTO cases (program, meeting_time, meeting_date) VALUES ('%s','%s','%s');'''
for c in cases:
print qry%(c.program, c.meeting_time, c.meeting_date)

Before I wrap up, one last thing - if you want to output this to a file instead of stdout
with open('cases.sql', 'w') as f:
qry = '''INSERT INTO cases (program, meeting_time, meeting_date) VALUES ('%s','%s','%s');'''
for c in cases:
print >>f, qry%(c.program, c.meeting_time, c.meeting_date)

Notice how I am redirecting print output to the file handle ‘f’. Here’s a blog article about it.

Sunday, July 11, 2010

Build mono-2.4.6 in Snow Leopard

Mono 2.4.6 configure scripts generates 32 bit build and requires 32 bit dependencies.

This is a problem in Snow Leopard which is all 64 bit. I had to use the following settings.


export CFLAGS="-I/Users/amitava/local/include -arch i386 -m32"
export CXXFLAGS="$CFLAGS"
export LDFLAGS="-L/Users/amitava/local/lib"

Please note that we have to override the system defaults by the generated packages. So I append $prefix (/Users/amitava/local in my case) to the PATH environment variable

export PATH=$HOME/local/bin:$PATH

The following packages were needed
gettext-0.18.1.1
libiconv-1.13.1
glib-2.12.12

And to build mono
export PKG_CONFIG_PATH=/Users/amitava/local/lib/pkgconfig:$PKG_CONFIG_PATH
./configure --prefix=$HOME/local --with-tls=pthread

Friday, June 25, 2010

Bash one-liner to build classpath

I found this gem in clojure-contrib's clj-env-dir
EXT="$(find -H $CLOJURE_EXT -mindepth 1 -maxdepth 1 -print0 | tr \\0 \:)"
Invoke java now as
java -cp $EXT clojure.main

Thursday, June 24, 2010

Java 6 api docs in MacOSX

If you are using Firefox, you can access the built-in java 6 api docs using the following url

jar:file:///System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/docs.jar!/docs/api/index.html

Wednesday, June 2, 2010

Friday, May 28, 2010

Title: How to upgrade maven in macosx?

See this at Nabble


In mac, maven is laid out as below


ashee-mac:~ amitava$ which mvn
/usr/bin/mvn
ashee-mac:~ amitava$ ls -l `which mvn`
lrwxr-xr-x 1 root wheel 24 May 25 09:55 /usr/bin/mvn -> /usr/share/maven/bin/mvn
ashee-mac:~ amitava$ ls -l /usr/share/maven
lrwxr-xr-x 1 root wheel 23 May 28 12:14 /usr/share/maven -> java/maven-2.2.0

So, all I needed is to download the latest version from apache, untar it under /usr/share/java
and update the symlink


ashee-mac:~ amitava$ unlink /usr/share/maven
ashee-mac:~ amitava$ ln -s /usr/share/java/apache-maven-2.2.1 /usr/share/maven

ashee-mac:~ amitava$ mvn --version
Apache Maven 2.2.1 (r801777; 2009-08-06 15:16:01-0400)
Java version: 1.6.0_20
Java home: /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
Default locale: en_US, platform encoding: MacRoman
OS name: "mac os x" version: "10.5.8" arch: "x86_64" Family: "mac"

Thursday, May 27, 2010

Scala Manifest

Idiomatic way to reflect on scala type


scala> def gt[A](x : A)(implicit m:Manifest[A]) = m
gt: [A](x: A)(implicit m: scala.reflect.Manifest[A])scala.reflect.Manifest[A]

scala> val l = 1 to 10
l: scala.collection.immutable.Range.Inclusive with scala.collection.immutable.Range.ByOne = Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> gt(l)
res8: scala.reflect.Manifest[scala.collection.immutable.Range.Inclusive with scala.collection.immutable.Range.ByOne] = scala.collection.immutable.Range$Inclusive with scala.collection.immutable.Range$ByOne

Thursday, May 6, 2010

Add user to groups in mac (leopard)


$ sudo dscl . create /groups/lms name lms
$ sudo dscl . create /groups/lms passwd "*"
$ sudo dscl . create /groups/lms gid 20014
$ sudo dscl . create /groups/lms users amitava


Related Links
http://osxfaq.com/tips/unix-tricks/week91/thursday.ws
http://paulbeachsblog.blogspot.com/2008/02/adding-user-to-group-using-dscl.html
http://support.apple.com/downloads/Server_Admin_Tools_10_5

Server Admin Tools does not allow me to edit stuff - just view it

Tuesday, April 20, 2010

VimClojure - REPL without opening a clojure file

If you just want to get a clojure REPL within vim using the excellent VimClojure, just change vim to clojure mode like so

:set ft=clojure

and start the REPL using typing \sr

Make sure, nailgun server is running in the background

Instructions to install VimClojure

Sunday, February 28, 2010

Friday, February 19, 2010

macosx tar - annoying ._* files

I noticed that tar generates files starting with ._* when creating an archive


$ tar cvzf jfa.tgz jfa


And the corresponding listing in archive

ashee-mac:shiva amitava$ tar tvf jfa.tgz
drwxr-xr-x amitava/staff 0 2010-02-19 09:18:22 jfa/
-rw-r--r-- amitava/staff 187 2010-02-19 08:40:06 jfa/._.htaccess
-rw-r--r-- amitava/staff 616 2010-02-19 08:40:06 jfa/.htaccess
....
....


Notice how it adds "._.htaccess"

I found the solution here

Here's the command for convenience
$COPYFILE_DISABLE=true tar cvzf jfa.tgz jfa



COPYFILE_DISABLE=true (for Leopard)
COPY_EXTENDED_ATTRIBUTES_DISABLE=true (for Tiger)

Monday, February 15, 2010

Clojure version

user=> *clojure-version*
{:major 1, :minor 1, :incremental 0, :qualifier ""}

Friday, February 5, 2010

F# in macosx

I downloaded FSharp-1.9.7.8 and attempted to run it in MacOSX using mono (2.4.2.3).


mono $FS_HOME/bin/fsi.exe


This kept exiting the prompt after brief attempts to spawn a GUI. I found the solution here

Invoke fsi.exe without gui and with readline as indicated in the link above


mono fsi.exe --readline+ --gui-

log4j - "appender is not assignable to Appender"

If you are getting "appender is not assignable to Appender" error in an attempt to configure log4j, just pass a system property as below

$) java -Dlog4j.ignoreTCL ...whatever...

See here

Wednesday, January 20, 2010

Hadoop 0.20.1 - Null pointer exception on startup

I am trying out Hadoop on my Mac0SX (10.5.8). I unzipped the distribution and after cd'ing to ~/hadoop-0.20.1/bin, ran


ashee-mac:bin amitava$ ./start-all.sh


I get the following exception

2010-01-20 13:14:13,341 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ashee-mac/10.17.8.21
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.1
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
************************************************************/
2010-01-20 13:14:13,480 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:175)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)

2010-01-20 13:14:13,482 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************



Turns out, the default does not come configured with, well "defaults". The fix is easy enough as highlighted in Quick Start (duh)

conf/core-site.xml:

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

conf/hdfs-site.xml:

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

conf/mapred-site.xml:

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>

Friday, January 15, 2010

OpenOffice - BootstrapException: no office executable found!

I was working on building (via mvn) an open office based tool for extracting images from powerpoint. But upon invocation I was getting the following


[amitava@lamp target]$ ./run.sh --extract --input pptsample.ppt --output-dir /tmp/foo
com.sun.star.comp.helper.BootstrapException: no office executable found!
at com.sun.star.comp.helper.Bootstrap.bootstrap(Bootstrap.java:253)
at edu.umich.med.umms.OpenOfficeUNODecomposition.main(OpenOfficeUNODecomposition.java:106)


Turns out, OO sdk wants you to reference the sdk jar's at the install location. I had to change the classpath like so


OO=/usr/lib64/openoffice.org/program/classes
CP=$CLASSPATH:./dcp-0.1.jar:$OO/juh.jar:$OO/jurt.jar:$OO/ridl.jar:$OO/unoil.jar
java -Djava.ext.dirs=./dependency -cp $CP edu.umich.med.umms.OpenOfficeUNODecomposition "$@"


This thread pointed me to the solution.

Thursday, January 14, 2010

Data Visualization

http://www.juicekit.org/
http://vis.stanford.edu/protovis/
http://manyeyes.alphaworks.ibm.com/manyeyes/
http://flare.prefuse.org/

Tuesday, January 12, 2010

Apache Tika (Lucene) build issue - java.lang.OutOfMemoryError: Java heap space

While building apache tika from source, I get the following issue


$ mvn package

[INFO] [bundle:bundle {execution: default-bundle}]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.(String.java:215)
at java.lang.StringBuilder.toString(StringBuilder.java:430)
at aQute.lib.osgi.Clazz.normalize(Clazz.java:696)
at aQute.lib.osgi.Clazz.doClassReference(Clazz.java:409)
at aQute.lib.osgi.Clazz.parseClassFile(Clazz.java:262)
at aQute.lib.osgi.Clazz.(Clazz.java:101)
at aQute.lib.osgi.Analyzer.analyzeJar(Analyzer.java:1647)
at aQute.lib.osgi.Analyzer.analyzeBundleClasspath(Analyzer.java:1563)
at aQute.lib.osgi.Analyzer.analyze(Analyzer.java:108)
at aQute.lib.osgi.Builder.analyze(Builder.java:294)
at aQute.lib.osgi.Builder.doConditional(Builder.java:260)
at aQute.lib.osgi.Builder.build(Builder.java:75)
at org.apache.felix.bundleplugin.BundlePlugin.buildOSGiBundle(BundlePlugin.java:391)
at org.apache.felix.bundleplugin.BundlePlugin.execute(BundlePlugin.java:282)
at org.apache.felix.bundleplugin.BundlePlugin.execute(BundlePlugin.java:236)
at org.apache.felix.bundleplugin.BundlePlugin.execute(BundlePlugin.java:227)
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:483)
......
......


The solution is easy enough - just increase memory like so

$ MAVEN_OPTS="-Xmx3g" mvn package


Hope this helps folks who are getting tripped on this issue

Tuesday, January 5, 2010

ocamlfind - macosx

While exploring ocaml, the system could not find ocamlfind in my macport install of ocaml.

A quick google search lead me to this post

All I had to do is the following

amitava:learn amitava$ sudo port install caml-batteries
---> Computing dependencies for caml-batteries
---> Fetching caml-findlib
---> Attempting to fetch findlib-1.2.5.tar.gz from http://distfiles.macports.org/caml-findlib
---> Verifying checksum(s) for caml-findlib
---> Extracting caml-findlib
---> Configuring caml-findlib
---> Building caml-findlib
---> Staging caml-findlib into destroot
---> Installing caml-findlib @1.2.5_0
---> Activating caml-findlib @1.2.5_0
---> Cleaning caml-findlib
---> Fetching caml-ounit
---> Attempting to fetch ounit-1.0.3.tar.gz from http://distfiles.macports.org/caml-ounit
---> Verifying checksum(s) for caml-ounit
---> Extracting caml-ounit
---> Configuring caml-ounit
---> Building caml-ounit
---> Staging caml-ounit into destroot
---> Installing caml-ounit @1.0.3_0
---> Activating caml-ounit @1.0.3_0
---> Cleaning caml-ounit
---> Fetching caml-type-conv
---> Attempting to fetch type-conv-1.6.10.tar.gz from http://www.ocaml.info/ocaml_sources
---> Verifying checksum(s) for caml-type-conv
---> Extracting caml-type-conv
---> Configuring caml-type-conv
---> Building caml-type-conv
---> Staging caml-type-conv into destroot
---> Installing caml-type-conv @1.6.10_0
---> Activating caml-type-conv @1.6.10_0
---> Cleaning caml-type-conv
---> Fetching caml-bin-prot
---> Attempting to fetch bin-prot-release-1.2.20.tar.gz from http://www.janestreet.com/ocaml
---> Verifying checksum(s) for caml-bin-prot
---> Extracting caml-bin-prot
---> Configuring caml-bin-prot
---> Building caml-bin-prot
---> Staging caml-bin-prot into destroot
---> Installing caml-bin-prot @1.2.20_0
---> Activating caml-bin-prot @1.2.20_0
---> Cleaning caml-bin-prot
---> Fetching caml-camlzip
---> Attempting to fetch camlzip-1.04.tar.gz from http://distfiles.macports.org/caml-camlzip
---> Verifying checksum(s) for caml-camlzip
---> Extracting caml-camlzip
---> Applying patches to caml-camlzip
---> Configuring caml-camlzip
---> Building caml-camlzip
---> Staging caml-camlzip into destroot
---> Installing caml-camlzip @1.04_0
---> Activating caml-camlzip @1.04_0
---> Cleaning caml-camlzip
---> Fetching caml-camomile
---> Attempting to fetch camomile-0.7.2.tar.bz2 from http://superb-east.dl.sourceforge.net/camomile
---> Attempting to fetch camomile-0.7.2.tar.bz2 from http://superb-west.dl.sourceforge.net/camomile
---> Attempting to fetch camomile-0.7.2.tar.bz2 from http://downloads.sourceforge.net/camomile
---> Verifying checksum(s) for caml-camomile
---> Extracting caml-camomile
---> Configuring caml-camomile
---> Building caml-camomile
---> Staging caml-camomile into destroot
---> Installing caml-camomile @0.7.2_0
---> Activating caml-camomile @0.7.2_0
---> Cleaning caml-camomile
---> Fetching caml-pcre
---> Attempting to fetch pcre-ocaml-6.0.1.tar.gz from http://www.ocaml.info/ocaml_sources/
---> Verifying checksum(s) for caml-pcre
---> Extracting caml-pcre
---> Applying patches to caml-pcre
---> Configuring caml-pcre
---> Building caml-pcre
---> Staging caml-pcre into destroot
---> Installing caml-pcre @6.0.1_0
---> Activating caml-pcre @6.0.1_0
---> Cleaning caml-pcre
---> Fetching caml-ocamlnet
---> Attempting to fetch ocamlnet-2.2.9.tar.gz from http://distfiles.macports.org/caml-ocamlnet
---> Verifying checksum(s) for caml-ocamlnet
---> Extracting caml-ocamlnet
---> Applying patches to caml-ocamlnet
---> Configuring caml-ocamlnet
---> Building caml-ocamlnet
---> Staging caml-ocamlnet into destroot
---> Installing caml-ocamlnet @2.2.9_0
---> Activating caml-ocamlnet @2.2.9_0
---> Cleaning caml-ocamlnet
---> Fetching caml-sexplib
---> Attempting to fetch release-4.2.16.tar.bz2 from http://distfiles.macports.org/caml-sexplib
---> Verifying checksum(s) for caml-sexplib
---> Extracting caml-sexplib
---> Configuring caml-sexplib
---> Building caml-sexplib
---> Staging caml-sexplib into destroot
---> Installing caml-sexplib @4.2.16_0
---> Activating caml-sexplib @4.2.16_0
---> Cleaning caml-sexplib
---> Fetching caml-batteries
---> Attempting to fetch batteries-20090405.tar.gz from http://distfiles.macports.org/caml-batteries
---> Verifying checksum(s) for caml-batteries
---> Extracting caml-batteries
---> Configuring caml-batteries
---> Building caml-batteries
---> Staging caml-batteries into destroot
---> Installing caml-batteries @20090405_0
---> Activating caml-batteries @20090405_0
---> Cleaning caml-batteries