Mikko Kortelainen

Running Claude Code Through GitHub Copilot

2026-03-31T06:00:00+03:00

Claude Code is Anthropic's CLI tool for working with Claude in the terminal. It's good for coding tasks, but it requires either a Claude subscription or an Anthropic API key.

GitHub copilot also supports Anthropic models. If you already have a GitHub Copilot subscription, there's a way to use those models with Claude Code. GitHub Copilot's API natively supports the Anthropic Messages format, so a thin local proxy can redirect Claude Code's requests to Copilot instead.

I wrote cc-gh-proxy for this. It's a single Python file with no dependencies beyond the standard library.

How It Works

The proxy sits between Claude Code and api.githubcopilot.com:

Claude Code --> localhost:4000 --> api.githubcopilot.com

All it does is:

Swap the auth header to a GitHub CLI OAuth token
Map model names (claude-opus-4-6 becomes claude-opus-4.6)
Strip cache_control fields that Copilot doesn't support

No format conversion is needed for the Anthropic Messages path. Token counts and streaming work natively.

Copilot also exposes an OpenAI-compatible endpoint, which is used for models other than Anthropic's, so the proxy translates those responses between OpenAI and Anthropic formats. This lets you use models like gpt-4 with Claude Code, albeit without streaming.

Setup

You need the GitHub CLI (gh) and a Copilot subscription:

gh auth login
gh auth refresh --hostname github.com -s copilot

Then point Claude Code at the proxy by adding this to .claude/settings.json in your project:

{
  "env": {
    "ANTHROPIC_BASE_URL": "http://localhost:4000"
  }
}

Start the proxy and run Claude Code as usual:

./cc-gh-proxy.py &
claude

Token Management

The proxy fetches an OAuth token from gh auth token at startup and refreshes it automatically every hour. If a request gets a 401, the token is refreshed immediately and the request retried. This is all thread-safe, so concurrent Claude Code sessions work fine.

Logging

The proxy writes structured request logs to logs/requests.jsonl -- one JSON object per request with model, messages, token usage, and timing.

Caveats

Copilot's API doesn't support Anthropic's prompt caching, so the cache_control fields are stripped. This means slightly higher latency since nothing is cached between requests.

This works today, but GitHub could change or restrict their Copilot API at any time. The proxy is experimental and may not support every edge case.

The log files may contain sensitive information, so be careful with them, and they also grow indefinitely. The proxy does not currently implement any log rotation or redaction.

The code is on GitHub: kortsi/cc-gh-proxy.

Moving from WordPress to Pelican

2026-03-17T12:00:00+02:00

After running this blog on WordPress since 2007, I've finally moved it to a static site. The trigger was my old hosting provider shutting down their data center at the end of March 2026. My server was getting old, too.

Rather than migrating WordPress somewhere else, I decided to go static. I also got a new domain (mikko.kortelainen.io) and a fresh coat of paint for the blog.

I have been lazy about writing new content for years, but I still wanted to keep the old articles and comments alive. The new stack is more maintainable and cost-effective, and I hope it will encourage me to write more often.

The New Stack

The blog now runs on Pelican, a Python-based static site generator. The infrastructure is on AWS, managed with CDK:

S3 for static file hosting
CloudFront for CDN and HTTPS
Lambda + API Gateway for the comment system
DynamoDB for storing comments and ratings

Articles and comments from 2007-2023 were migrated from WordPress, and the old domain redirects to the new one.

I wanted to keep everything that mattered from the old site:

All articles and their content
All comments with full threading
Article ratings migrated from the WordPress Post Ratings plugin
Old domain URLs redirect to the new domain, including the date-based WordPress URL scheme (/blog/2015/03/20/slug/ becomes /blog/slug/)
RSS feed subscribers are redirected to the new Atom feed

Some articles were missing content for an unknown reason, and I was not able to recover everything.

The comment system is custom-built: a Lambda function behind API Gateway with Akismet spam filtering, per-IP rate limiting, and optional Google sign-in.

The new design is minimalist and mobile-friendly, with a light/dark mode toggle.

Let me know if you find any issues with the new site or have suggestions for improvements.

Enabling Quectel EM05-G Modem on Linux

2022-12-21T22:55:00+02:00

My new Thinkpad has a Quectel EM05-G modem. For some reason it refused to connect, even though it showed up in Gnome settings on Ubuntu 22.10. I checked with mmcli -m 0 and it said sim missing as its state.

After some Googling it appeared the modem ships in some kind of a locked state, so it only works in Windows. I checked with "mbimcli":

$ sudo apt install libmbim-utils
$ sudo mbimcli -p -d /dev/cdc-wdm0 --quectel-query-radio-state
[/dev/cdc-wdm0] Radio state retrieved: 'fcc-locked'

It was easy to unlock:

$ sudo mbimcli -p -d /dev/cdc-wdm0 --quectel-set-radio-state=on
[/dev/cdc-wdm0] Successfully requested to enable radio
$ sudo mbimcli -p -d /dev/cdc-wdm0 --quectel-query-radio-state
[/dev/cdc-wdm0] Radio state retrieved: 'on'

After this, Gnome was able to enable the mobile broadband connection without problem.

Tunneling SSH over HTTPS with stunnel

2020-02-06T15:56:00+02:00

I was faced with a firewall denying access to the outside world using ssh. All I had was http/https access via a proxy server which required authentication. I had an Ubuntu jump host outside the network connected to the internet with a free 443 port. I tried accessing that with httptunnel and proxytunnel, but could get neither to work with this proxy server.

The solution that worked in this particular case was stunnel. It can wrap any TCP connection into an https session which was not rejected by the proxy server I was facing.

Client

Install stunnel on laptop (mine is Ubuntu 18.04), and use openssl to generate a key and a certificate:

sudo su -
apt-get install stunnel4
cd /etc/stunnel
openssl genrsa 1024 > stunnel.key
openssl req -new -key stunnel.key -x509 -days 1000 -out stunnel.crt
# (you can accept openssl defaults)
cat stunnel.crt stunnel.key >stunnel.pem

The values you enter in the client certificate request do not matter.

Next, create stunnel configuration. You should replace the variables with your own values.

PROXY_ADDRESS="<your proxy_address:proxy_port>"
PROXY_USERNAME="<your proxy username>"
PROXY_PASSWORD="<your proxy password>"
JUMP_HOST_ADDRESS="<your jump host address>"

cat >/etc/stunnel/stunnel.conf <<EOF
pid=/var/run/stunnel.pid
cert=/etc/stunnel/stunnel.pem

[ssh]
client=yes
libwrap=no
accept=2222
connect=$PROXY_ADDRESS
protocolUsername=$PROXY_USERNAME
protocolPassword=$PROXY_PASSWORD
protocolHost=$JUMP_HOST_ADDRESS:443
EOF

systemctl start stunnel4

Stunnel should now be running on your laptop. It will forward any connection you make to localhost:2222 to your $JUMP_HOST port 443 and wrap it in an https request.

Jump host

The proxy server did not accept self-signed certificates, so I installed certbot to get a working certificate on the jump host (mine is Ubuntu 19.10):

sudo su -
add-apt-repository ppa:certbot/certbot
apt-get install certbot
certbot certonly --standalone

# Make note where certbot installs the private key
# and the full certificate chain

Install stunnel on server and configure it:

JUMP_HOST_DNS_NAME="<your jump host DNS name>"

apt-get install stunnel4

cat >/etc/stunnel/stunnel.conf <<EOF
pid=/var/run/stunnel.pid
key=/etc/letsencrypt/live/$JUMP_HOST_DNS_NAME/privkey.pem
cert=/etc/letsencrypt/live/$JUMP_HOST_DNS_NAME/fullchain.pem

[ssh]
accept=443
connect=127.0.0.1:22
EOF

systemctl start stunnel4

Make sure the key and cert paths are pointing to correct certbot files.

Stunnel should now be running on the jump host as well. It will listen on port 443, unwrap the https and redirect the connection to localhost port 22, where sshd should be listening.

Test From Client

Now you should be able to ssh to your jump host by connecting to the local stunnel instance:

ssh localhost -p 2222

Methods for Both Classes and Instances in Python

2017-07-24T12:28:00+03:00

I came across a situation where it would be nice for a Python class to have a method which works for both the class and its instances, and when called in an instance context, to know on which instance it was invoked. Python does not support this directly, but as it is a very flexible language, it can be done. To make it work somewhat nicely requires just a few lines of not so obvious code.

To have Python call a method in the context of a class requires the use of the decorator @classmethod. You can call such a method for an instance as well, but the method is only passed the class as an argument, not the instance.

Here's a an example:

class X:
    @classmethod
    def m(cls, *args, **kwargs):
        print('cls:    {}'.format(cls))
        print('self:   {}'.format('NO SELF!'))
        print('args:   {}'.format(args))
        print('kwargs: {}'.format(kwargs))

The class has one classmethod which simply prints what it knows about the world around it. It knows the class and the arguments passed to it. When invoked on a class instance, it does not know which instance it was invoked on.

In [2]: X.m('arg', kwarg='kwarg')
cls:    <class '__main__.X'>
self:   NO SELF!
args:   ('arg',)
kwargs: {'kwarg': 'kwarg'}

In [3]: x=X()

In [4]: x.m('arg', kwarg='kwarg')
cls:    <class '__main__.X'>
self:   NO SELF!
args:   ('arg',)
kwargs: {'kwarg': 'kwarg'}

What I needed was to have self defined when invoked in an instance context. I solved it by injecting self into the classmethod when the class constructor is run at instantiation time. This requires a slight modification to the classmethod's signature, and replacing the original classmethod with an instancemethod which takes both the class as a positional argument and the instance as a keyword argument. This is achieved nicely with a utility function called class2instancemethod:

from types import MethodType

def class2instancemethod(method, self):
    """Converts a classmethod into an instancemethod"""
    def class2instance(*args, **kwargs):
        # Remove first positional arg which is self
        # and move it to kwargs
        return method(*(args[1:]), self=self, **kwargs)
    return MethodType(class2instance, self)

class X:
    def __init__(self):
        self.m = class2instancemethod(self.m, self)

    @classmethod
    def m(cls, *args, self=None, **kwargs):
        print('cls:    {}'.format(cls))
        print('self:   {}'.format(self))
        print('args:   {}'.format(args))
        print('kwargs: {}'.format(kwargs))

There are four things to note here. The first one is the class2instancemethod utility function which takes a classmethod and a class instance as argument and returns an instancemethod which has class ("cls") as the first positional parameter and instance ("self") as a keyword argument.

The second interesting thing is the class constructor which calls our utility function at class instantiation time and replaces the original classmethod with our patched version.

The third interesting thing is the signature of the classmethod m:

@classmethod
def m(cls, *args, self=None, **kwargs):

Argument list now contains self as a keyword argument. When invoked in a class context, self will be None. When invoked in an instance context, self will receive the instance.

The fourth thing to note is the print line:

print('self:   {}'.format(self))

We now have a self to print. Here's what looks like now:

In [6]: X.m('arg', kwarg='kwarg')
cls:    <class '__main__.X'>
self:   None
args:   ('arg',)
kwargs: {'kwarg': 'kwarg'}

In [7]: x=X()

In [8]: x.m('arg', kwarg='kwarg')
cls:    <class '__main__.X'>
self:   <__main__.X object at 0x7f1fe54250b8>
args:   ('arg',)
kwargs: {'kwarg': 'kwarg'}

So now it is possible to have the classmethod know the instance when called in an instance context.

With this implementation we would have to patch every method in our constructor method, which is repetitive and error-prone.

We can improve the situation by making our own decorator for such methods, and a helper function which needs to be called just once in the constructor. Here's the full code:

from types import MethodType

def class2instancemethod(method, self):
    """Converts a classmethod into an instancemethod"""
    def class2instance(*args, **kwargs):
        # Remove first positional arg which is self
        # and move it to kwargs
        return method(*(args[1:]), self=self, **kwargs)
    return MethodType(class2instance, self)

_c2im_attr = '_convert_to_instancemethod'

def instancemethod(m):
    """Decorator for classmethods which should be
    converted to instancemethods in constructor.
    You must run make_instancemethods(self) in your
    constructor to actually make it happen.

    Eg:

    class X:
        @classmethod
        @instancemethod
        def m1(cls, *args, self=None, **kwargs):
            # - cls is always set
            # - self is set when invoked on an instance
            # - args & kwargs stay the same for both ways

            print('cls:    {}'.format(cls))
            print('self:   {}'.format(self))
            print('args:   {}'.format(args))
            print('kwargs: {}'.format(kwargs))

            if self is not None:
                print('Called on instance {}'.format(self))
            else:
                print('Called on class {}'.format(cls))

        @instancemethod
        @classmethod
        def m2(cls, *args, self=None, **kwargs):
            print('self:   {}'.format(self))

        def __init__(self):
            # Changes all classmethods marked with
            # @instancemethod so that self is injected
            # as kwarg
            self.make_instancemethods(self)
    """
    if type(m)!=classmethod:
        # Set flag
        setattr(m, _c2im_attr, True)
    else:
        # Dig out the original method and set flag there
        setattr(m.__func__, _c2im_attr, True)
    return m

def make_instancemethods(self):
    """Takes all classmethods decorated with the
    @instancemethod decorator and injects self
    as a kwarg."""
    c2im = [x for x in dir(self)
            if not x.startswith('__')]

    for m in c2im:
        method = getattr(self, m)
        if getattr(method, _c2im_attr, False):
            setattr(self, m,
                    class2instancemethod(method, self))

And here's an example on how to use it:

class X:
    def __init__(self):
        make_instancemethods(self)

    @classmethod
    @instancemethod
    def m1(cls, *args, self=None, **kwargs):
        print('cls:    {}'.format(cls))
        print('self:   {}'.format(self))
        print('args:   {}'.format(args))
        print('kwargs: {}'.format(kwargs))

    @instancemethod
    @classmethod
    def m2(cls, arg1=1, self=None):
        print('self:   {}'.format(self))
        print('arg1:   {}'.format(arg1))

Example output:

In [11]: X.m1('arg', kwarg='kwarg')
cls:    <class '__main__.X'>
self:   None
args:   ('arg',)
kwargs: {'kwarg': 'kwarg'}

In [12]: X.m2()
self:   None
arg1:   1

In [13]: X.m2(2)
self:   None
arg1:   2

In [14]: x=X()

In [15]: x.m1('arg', kwarg='kwarg')
cls:    <class '__main__.X'>
self:   <__main__.X object at 0x7fcf7c9c5550>
args:   ('arg',)
kwargs: {'kwarg': 'kwarg'}

In [16]: x.m2()
self:   <__main__.X object at 0x7fcf7c9c5550>
arg1:   1

In [17]: x.m2(2)
self:   <__main__.X object at 0x7fcf7c9c5550>
arg1:   2

This makes it quite easy to have classmethods which are able to get the target object when invoked on an instance.

The only tricky thing is to have the arguments in the right order. The order should be this:

cls
any positional arguments
*args
any named keyword arguments you wish to be able to use also as positional ones (when not using *args)
self=None
*kwargs

Only cls and self=None are required, the rest are optional.

As a final note I would like to point out that this has only been tested with Python 3.5.

Enable vi editing mode in IPython 5

2016-08-04T12:46:00+03:00

Create default profile and set editing mode to 'vi':

ipython profile create
echo "c.TerminalInteractiveShell.editing_mode = 'vi'" \
  >>~/.ipython/profile_default/ipython_config.py

Keeping SSH Tunnels Up With Autossh

2015-11-27T17:05:00+02:00

To keep an ssh connection with a tunnel for port forwarding up reliably we can use the autossh command by Carson Harding. If the connection drops, autossh will restart it. Here's a quick recipe to forward local port 33306 to a remote MySQL host listening on port 3306:

# Install autossh
sudo apt-get install autossh

# Make a user and su to it
sudo useradd -m -s /bin/false autossh
sudo su - autossh -s /bin/bash

# Our connection parameters
USERNAME="sshuser"
DESTINATION="192.168.1.1"
FORWARD="33306:127.0.0.1:3306"

# Make an ssh identity, copy to remote
ssh-keygen
ssh-copy-id $USERNAME@$DESTINATION

# Set keepalive options
cat >~/.ssh/config <<'EOF'
ServerAliveInterval 30
ServerAliveCountMax 5
EOF

# To start foreground:
autossh -M 0 $DESTINATION -l $USERNAME -L $FORWARD
exit

# To start background:
autossh -f -M 0 -N -o BatchMode=yes $DESTINATION -l $USERNAME -L $FORWARD

# To stop:
killall autossh

# Leave su
exit

# Start background as autossh
USERNAME="sshuser"
DESTINATION="192.168.1.1"
FORWARD="33306:127.0.0.1:3306"
sudo su - autossh -s /bin/bash -c "autossh -f -M 0 -N -o BatchMode=yes $DESTINATION -l $USERNAME -L $FORWARD"

We create a local user which will be used to run the autossh command. We also create a new ssh key pair and copy the public key to the remote user account on the destination host. The ServerAlive options will make ssh test the connection periodically, in this case every 30 seconds. If 5 consecutive tests fail, ssh will terminate. Autossh will then restart the connection.

Autossh itself can be run in the foreground, when it will behave like a regular ssh session, or in the background, which is especially useful for keeping up ssh tunnels for port forwarding. The -f option will make autossh run in the background. The -M 0 option disables the monitoring channel which we do not need since ssh will exit when it detects a dropped connection. The -N option of ssh causes ssh not to run any command, not even a shell in the session. It just starts your tunnels, which is what we need. The -o BatchMode option will disable password authentication.

To autostart at reboot, add these lines to /etc/rc.local:

USERNAME="sshuser"
DESTINATION="192.168.1.1"
FORWARD="33306:127.0.0.1:3306"
su - autossh -s /bin/bash -c "autossh -f -M 0 -N -o BatchMode=yes $DESTINATION -l $USERNAME -L $FORWARD"

Remember to add it before the exit line if there is one.

This thing has been tested on Ubuntu 14.04.

Flask-SQLAlchemy and PostgreSQL Unit Testing with Transaction Savepoints

2015-10-22T23:04:00+03:00

PostgreSQL provides us two very powerful features which are helpful with unit testing: transactional DDL and transaction savepoints. In this article I will show you how to use those with Flask-SQLAlchemy unit tests.

Transactional DDL means you can create tables inside a transaction, run tests against them, and roll back your changes after you are done. The database will be left in the same state as it was when you started. If you started with an empty database, nothing will be left in the database afterwards.

Savepoints allow you to roll back a part of a transaction without affecting what has happened before that savepoint was created. You can run a single test inside a savepoint and roll back just the changes that single test made without affecting the changes your set-up code has done before the savepoint.

That means you can create a large number of tables and other database objects in the beginning of your test suite and then run individual tests inside nested transaction using savepoints. There is no need to drop and re-create the whole database schema for each test.

Table of Contents

1 PostgreSQL Database for Unit Tests
2 Other Requirements
3 An App, a Model, and Some Tests
4 Tests in Nested Transactions
5 Transactional DDL
6 Abstract it Out
7 Download the Code

1 PostgreSQL Database for Unit Tests

We will need an empty database to run our unit tests against. If you have PostgreSQL already installed, at least on Ubuntu there are command line tools to get it done:

postgres@host:~$ createuser -P unittestuser
Enter password for new role: password
Enter it again: password

postgres@host:~$ createdb -O unittestuser unittestdb

The createuser command creates a database user. The createdb creates the database, and the -O option sets the owner of the database.

2 Other Requirements

You obviously need to install Flask-SQLAlchemy and psycopg2 for PostgreSQL connectivity:

(virtualenv)user@host:~$ pip install flask flask-sqlalchemy psycopg2

3 An App, a Model, and Some Tests

First let's take a look at an ordinary test suite. Flask-SQLAlchemy wraps SQLAlchemy and integrates it with Flask, providing some additional functionality with it. A very minimal Flask app with Flask-SQLAlchemy and PostgreSQL support could look something like below.

Please note that I haven't tested the following example nor have I run its tests. The downloadable file at the end of this article contains a working and tested example, and includes the tests used to test it. The example below gives us something we can start do discuss here. Let me know if you find errors.

from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql+psycopg2://unittestuser:password@localhost/unittestdb'
db = SQLAlchemy(app)

class MyModel(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String)

    def update_name(self, new_name):
        self.name = new_name

Our app is in deed very minimal. It doesn't even have any views. But it is enough for our purposes.

A minimal test suite for our database model might look like this:

from unittest import TestCase
from app import db, MyModel

class MyModelTestCase(TestCase):
    def setUp(self):
        db.create_all()
        db.session.add(MyModel(name='Test'))
        db.session.commit()

    def tearDown(self):
        db.drop_all()

    def test_model_exists(self):
        """Model should exist"""
        self.assertTrue(
            db.session.query(MyModel).filter(MyModel.name=='Test').first()
            is not None)

    def test_model_update(self):
        """Model modification should modify model"""
        model = db.session.query(MyModel).filter(MyModel.name=='Test').first()
        model.update_name('New name')
        db.session.commit()
        self.assertTrue(
            db.session.query(MyModel).filter(MyModel.name=='New name').first()
            is not None)

Note that the db.create_all/drop_all is executed for each test, as well as the initial model adding. We can avoid this by using a transaction and some savepoints.

4 Tests in Nested Transactions

So how do we use savepoints to our advantage when writing unit tests for Flask-SQLAlchemy apps? To test arbitrary Flask-SQLAlchemy code inside a nested transaction, we must replace the global db.session object with one that runs in a nested transaction. After all, your code is likely to perform all of its database operations through the db.session object.

We can choose to use either the TestCase class's setUpClass/tearDownClass classmethods which are called once per each TestCase, or the setUpModule/tearDownModule functions which unittest will call once per module when running the tests. And we can even use both if we want to do some setup work at the module level and some more in each class.

Here's an example of how to do it using the classmethods:

class MyModelTestCase(TestCase):
    @classmethod
    def setUpClass(cls):
        # Create schema
        db.create_all()
        db.session.add(MyModel(name='Test'))
        db.session.commit()

    @classmethod
    def tearDownClass(cls):
        # Drop schema
        db.drop_all()

    def setUp(self):
        # Create two savepoint
        self.savepoint1 = db.session.begin_nested()
        self.savepoint2 = db.session.begin_nested()

        # Make backup of session and replace with savepoint
        self.session_backup = db.session
        db.session = self.savepoint2.session

    def tearDown(self):
        # Roll back to first savepoint
        self.savepoint1.rollback()

        # Restore original session
        db.session = self.session_backup

    def test_models_exists(self):
        """Models should exist"""
        self.assertTrue(
            db.session.query(MyModel).filter(MyModel.name=='Test').first()
            is not None)

    def test_model_update(self):
        """Model modification should modify model"""
        model = db.session.query(MyModel).filter(MyModel.name=='Test').first()
        model.update_name('New name')
        db.session.flush()
        self.assertTrue(
            db.session.query(MyModel).filter(MyModel.name=='New name').first()
            is not None)

The setUpClass method is only run once in the beginning of the test. The database schema is created there. The schema is dropped in the end in the tearDownClass method.

The setUp method now creates two savepoints. Strictly speaking only one savepoint is needed, but I would recommend two because the code you are testing might call db.session.commit() and that will commit the savepoint it is running in. A committed savepoint can not be rolled back to. So in the tearDown method we will roll back to the first savepoint which will be available also in the case the tested code commits. When the savepoint is rolled back, the next test case will see the database exactly like it was after the setUpClass call.

Note that if your code does multiple commits per test, this doesn't work and you will receive exceptions from SQLAlchemy. You can add more savepoints to work around that. You must have one more savepoint than you have commits in a single test. On the other hand, it is a good way of finding code paths that perform multiple commits. Doing lots of commits will result in a lot of disk activity and will slow down your application.

If you want to do multiple commits in your test case, you can use db.session.begin_nested and commit that:

class MyModelTestCase(TestCase):
    def test_with_commits(self):
        savepoint1 = db.session.begin_nested()
        ... do stuff ...
        savepoint1.session.add(stuff)
        savepoint1.commit()
        ... assert ...

        savepoint2 = db.session.begin_nested()
        ... do more stuff ...
        savepoint2.session.add(stuff)
        savepoint2.commit()
        ... assert some more ...

Stuff you do inside a committed savepoint will be available later on, so the savepoint1 stuff in the example above will be available inside savepoint2, unless you choose to roll back to savepoint1 instead of committing. Eventually of course the test tearDown will roll back to its own savepoint to undo everything you did before running the next test.

5 Transactional DDL

The next logical step is to run db.create_all() inside the same transaction as the tests. This way the database you run unit tests against will start empty and end up empty without even the need to drop tables afterwards.

In fact it might even be possible to run it against a database which has existing tables already if you drop first, and keep your old data after the tests. But I haven't tested if it works or not. It is probably not a good idea, though. There might be side effects since some database objects, such as sequences, are not kept completely isolated inside the transaction.

Flask-SQLAlchemy does not allow us to run the metadata creation inside the same transaction as the rest of the app. Again we need to monkey patch the db object and replace the session with our own, this time to make a new database connection. Flask-SQAlchemy depends on the db.session object to be a factory for new sessions, but we want to avoid new sessions since we want everything to run inside a single transaction. We will have to use a custom session which is a "factory" that returns itself.

Below is an example of how to do it in the setUpModule/tearDownModule functions.

from flask.ext.sqlalchemy import SessionBase

class TestingSession(SessionBase):
    def __init__(self, db, bind, **options):
        self.app = db.get_app()

        SessionBase.__init__(
            self, autocommit=False, autoflush=True,
            bind=bind, **options
        )

    def __call__(self):
        # Flask-SQLAlchemy wants to create a new session
        # Simply return the existing session
        return self

    def get_bind(self, mapper=None, clause=None):
        # mapper is None if someone tries to just get a connection
        if mapper is not None:
            info = getattr(mapper.mapped_table, 'info', {})
            bind_key = info.get('bind_key')
            if bind_key is not None:
                state = flask.ext.sqlalchemy.get_state(self.app)
                return state.db.get_engine(self.app, bind=bind_key)
        return SessionBase.get_bind(self, mapper, clause)

from unittest import TestCase
from app import db

def setUpModule():
    global session_backup

    # Create a connection and start a transaction. This is needed so that
    # we can run the drop_all/create_all inside the same transaction as
    # the tests
    connection = db.engine.connect()
    transaction = connection.begin()

    # Back up the original session and replace with our own
    session_backup = db.session
    db.session = TestingSession(db, connection)

    ## Drop all to get an empty database free of old crud just in case
    db.metadata.drop_all(transaction.connection)

    ## Create everything
    db.metadata.create_all(transaction.connection)

def tearDownModule():
    # Roll back everything
    db.session.rollback()

    # Restore backup
    db.session = session_backup

class MyModelTestCase(TestCase):
    @classmethod
    def setUpClass(cls):
        # Create a model instance for this class
        db.session.add(MyModel(name='Test'))
        db.session.flush()

        # Create class savepoint
        cls.savepoint = db.session.begin_nested()

        # Backup and replace
        cls.session_backup = db.session
        db.session = self.savepoint

    @classmethod
    def tearDownClass(cls):
        # Roll back to class savepoint
        cls.savepoint.rollback()

        # Restore original session
        db.session = cls.session_backup

    def setUp(self):
        # Create two test savepoints
        self.savepoint1 = db.session.begin_nested()
        self.savepoint2 = db.session.begin_nested()

        # Make backup of session and replace with savepoint
        self.session_backup = db.session
        db.session = self.savepoint2.session

        # This is for using app_context().pop()
        db.session.remove = lambda: None

    def tearDown(self):
        # Roll back to first test savepoint
        self.savepoint1.rollback()

        # Restore original session
        db.session = self.session_backup
    .
    .
    .

What setUpModule does is it makes its own database connection instead of using the one Flask-SQLAlchemy has created. This is needed so that we can begin a transaction, run the DDL in it, and continue with running the tests in the same transaction. The Flask-SQLAlchemy db.create_all() will commit the schema into the database if we use it directly. The TestingSession is a modified Flask-SQLAlchemy SignallingSessionclass. It keeps everything running in our single transaction.

Another detail to note is the db.session.remove = lambda: None statement in the test setUp code. It is a required fix when you push the application context in your own setUp method and pop it in the tearDown method. Flask-SQLAlchemy will raise an exception during app context teardown without it.

The rest of it is a simple extension of the original idea. Here we have two levels of setup code instead of one, and savepoints to guard them both. The setUpModule call creates the schema once for the whole module, while the model creation is done in the setUpClass method. It means that if you add another TestCase in the same module the schema will stay intact but the model inserted or any other operation performed in MyModelTestCase's setUpClass will only be there for MyModelTestCase's tests.

6 Abstract it Out

It is obviously not a good idea to keep code like that in each test module so let's extract the functionality and put it in a separate module which we will call flask_sqlalchemy_testing. The code for this and the tests can be downloaded at the end of the article.

from flask.ext.sqlalchemy import SessionBase

class TestingSession(SessionBase):
    def __init__(self, db, bind, **options):
        self.app = db.get_app()

        SessionBase.__init__(
            self, autocommit=False, autoflush=True,
            bind=bind, **options
        )

    def __call__(self):
        # Flask-SQLAlchemy wants to create a new session
        # Simply return the existing session
        return self

    def get_bind(self, mapper=None, clause=None):
        # mapper is None if someone tries to just get a connection
        if mapper is not None:
            info = getattr(mapper.mapped_table, 'info', {})
            bind_key = info.get('bind_key')
            if bind_key is not None:
                state = flask.ext.sqlalchemy.get_state(self.app)
                return state.db.get_engine(self.app, bind=bind_key)
        return SessionBase.get_bind(self, mapper, clause)

from unittest import TestCase

from app import db

def setUpModule():
    """Sets up the module so that we can run the whole test suite inside
    a single transaction. Each test will be executed in a nested transaction
    using PostgreSQL savepoints.

    We will:

    1. Start a transaction
    2. Drop everything
    3. (Re-)create everything
    """
    global session_backup, connection

    # Create a connection and start a transaction. This is needed so that
    # we can run the drop_all/create_all inside the same transaction as
    # the tests
    connection = db.engine.connect()
    transaction = connection.begin()

    # Back up the original session and replace with our own
    session_backup = db.session
    db.session = TestingSession(db, connection)

    ## Drop all to get an empty database free of old crud
    db.metadata.drop_all(transaction.connection)

    ## Create everything
    db.metadata.create_all(transaction.connection)

def tearDownModule():
    """Roll back everything, leaving database as it was before we started"""
    db.session.rollback()
    connection.close()

    # Restore backup
    db.session = session_backup

class FlaskDbTestCase(TestCase):
    @classmethod
    def setUpClass(cls):
        """Makes a savepoint for this class"""
        cls.savepoint1 = db.session.begin_nested()
        cls.savepoint2 = db.session.begin_nested()

        # Replace app db object temporarily with a nested tx
        cls.session_backup = db.session
        db.session = cls.savepoint2.session

    @classmethod
    def tearDownClass(cls):
        """Rolls back to the class savepoint"""
        cls.savepoint1.rollback()

        # Restore original session
        db.session = cls.session_backup

    def setUp(self):
        """Makes a savepoint for this test"""
        # We actually need to make 2 savepoints in case the code to be tested
        # happens to issue a commit so that we can roll back to this point.
        # A commit in the tested code can only commit "savepoint2" so we
        # will use "savepoint" for the rollback.
        self.savepoint1 = db.session.begin_nested()
        self.savepoint2 = db.session.begin_nested()

        # Replace app db object temporarily with a nested tx
        self.session_backup = db.session
        db.session = self.savepoint2.session

        # This is for using app_context().pop()
        db.session.remove = lambda: None

    def tearDown(self):
        """Rolls back to the test class savepoint"""
        # Roll back savepoint - schema will back to fresh
        self.savepoint1.rollback()

        # Restore original session
        db.session = self.session_backup

Let's use an app which has two models for a change:

from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql+psycopg2://unittestuser:password@localhost/unittestdb'
db = SQLAlchemy(app)

class ModelX(db.Model):
    x = db.Column(db.Integer, primary_key=True)

class ModelY(db.Model):
    y = db.Column(db.Integer, primary_key=True)

Here's the test module:

import flask_sqlalchemy_testing
from flask_sqlalchemy_testing import FlaskDbTestCase

from app import db, ModelX, ModelY

def setUpModule():
    flask_sqlalchemy_testing.setUpModule()

    # You can issue module-wide setup operations here
    # (these will be rolled back after all tests in this module have run)
    db.session.add(ModelX(x=0))
    db.session.add(ModelY(y=0))
    db.session.commit()

from flask_sqlalchemy_testing import tearDownModule

class ModelXTestCase(FlaskDbTestCase):
    @classmethod
    def setUpClass(cls):
        super(ModelXTestCase, cls).setUpClass()

        # These will be rolled back after all the tests in this class
        db.session.add(ModelX(x=1))
        db.session.flush()

    def setUp(self):
        super(ModelXTestCase, self).setUp()

        # These will be rolled back after each test
        db.session.add(ModelX(x=2))
        db.session.flush()

    def test_model_x_instance_0_exists(self):
        """Model X instance 0 should exist"""
        self.assertIsNotNone(
            ModelX.query.filter_by(x=0).first())

    def test_model_x_instance_1_exists(self):
        """Model X instance 1 should exist"""
        self.assertIsNotNone(
            ModelX.query.filter_by(x=1).first())

    def test_model_x_instance_2_exists(self):
        """Model X instance 2 should exist"""
        self.assertIsNotNone(
            ModelX.query.filter_by(x=2).first())

    def test_model_y_instance_0_exists(self):
        """Model Y instance 0 should exist also in ModelXTestCase"""
        self.assertIsNotNone(
            ModelY.query.filter_by(y=0).first())

    def test_model_y_instance_1_exists_not(self):
        """Model Y instance 1 should not exist in ModelXTestCase"""
        self.assertIsNone(
            ModelY.query.filter_by(y=1).first())

    def test_model_y_instance_2_exists_not(self):
        """Model Y instance 2 should not exist in ModelXTestCase"""
        self.assertIsNone(
            ModelY.query.filter_by(y=2).first())

    def test_savepoint(self):
        """Test model states with savepoint"""
        from sqlalchemy import inspect
        modelx = ModelX(x=3)
        model_state = inspect(modelx)
        self.assertTrue(model_state.transient)

        savepoint = db.session.begin_nested()
        savepoint.session.add(modelx)
        self.assertTrue(model_state.pending)

        savepoint.commit()
        self.assertTrue(model_state.persistent)

class ModelYTestCase(FlaskDbTestCase):
    @classmethod
    def setUpClass(cls):
        super(ModelYTestCase, cls).setUpClass()

        db.session.add(ModelY(y=1))
        db.session.flush()

    def setUp(self):
        super(ModelYTestCase, self).setUp()

        db.session.add(ModelY(y=2))
        db.session.flush()

    def test_model_y_instance_0_exists(self):
        """Model Y instance 0 should exist"""
        self.assertIsNotNone(
            ModelY.query.filter_by(y=0).first())

    def test_model_y_instance_1_exists(self):
        """Model Y instance 1 should exist"""
        self.assertIsNotNone(
            ModelY.query.filter_by(y=1).first())
        self.assertTrue(
            db.session.query(ModelY).filter(ModelY.y==1).first()
            is not None)

    def test_model_y_instance_2_exists(self):
        """Model Y instance 2 should exist"""
        self.assertIsNotNone(
            ModelY.query.filter_by(y=1).first())

    def test_model_x_instance_0_exists(self):
        """Model X instance 0 should exist also in ModelYTestCase"""
        self.assertIsNotNone(
            ModelX.query.filter_by(x=0).first())

    def test_model_x_instance_1_exists_not(self):
        """Model X instance 1 should not exist in ModelYTestCase"""
        self.assertIsNone(
            ModelX.query.filter_by(x=1).first())

    def test_model_x_instance_2_exists_not(self):
        """Model X instance 2 should not exist in ModelYTestCase"""
        self.assertIsNone(
            ModelX.query.filter_by(x=2).first())

if __name__ == '__main__':
    #import logging
    #log = logging.getLogger(__name__)
    #logging.basicConfig(level=logging.INFO)
    #logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)

    import unittest
    unittest.main(verbosity=2)

You can run the tests with "python test_app.py". You can uncomment the lines which enable sqlalchemy.engine logging to see the SQL it is issuing.

We must of course call the setUpModule of the flask_sqlalchemy_testing module in the start of our own setUpModule. If you don't need to override it, remember to import it. The tearDownModule also needs to be imported so that it will run. The same goes for setUpClass/tearDownClass.

7 Download the Code

The code is available to download and use as you wish. Here it is:

flask_sa_testing.tar.gz

I have used it, but not very extensively yet. Let me know if you find errors in it.

Here are the commands to run it in a virtualenv:

tar zxvf flask_sa_testing
cd flask_sa_testing
virtualenv virtualenv
source virtualenv/bin/activate
pip install -r requirements.txt
createuser -P unittestuser
createdb -O unittestuser unittestdb
python test_app.py

Ubuntu 14.04 Active Directory Authentication

2015-06-16T17:33:00+03:00

In a post a couple of years ago I gave an example on how to configure an Ubuntu 12.04 server to authenticate to Active Directory. Things used to be hard back then. Now we have the realmd realm enrollment manager to do the hard work of joining the host to an Active Directory domain, and the System Security Services Daemon or SSSD to do the actual authentication and authorization work whenever it is needed. And things are much easier to configure and get running.

Also in the mean time Microsoft has deprecated the Identity Management for UNIX extension to Active Directory. It used to be used to manage POSIX attributes in the AD for use by UNIX clients. Luckily, the SSSD has a nice coherent way of mapping Windows user and group ids to UNIX ones so that POSIX attributes may not be needed at all in the AD anymore, making things more straighforward. If you still need to be able configure attributes by individual LDAP entry basis, you may need to look into FreeIPA and ID Views. The automatic id mapping is not compatible with the old POSIX attributes in the sense that once you enable automatic id mapping, all the existing POSIX attributes are ignored. So you may have to fix group memberships, for example, if your POSIX group memberships don't match the Windows group memberships (the Windows group memberships are the ones that will be used with id mapping). And of course all the uid numbers will be changed when you flip the switch and enable automatic id mapping.

If you haven't been using POSIX attributes in the AD schema before, you don't have to worry about anything I said in the last paragraph. It just works.

Table of Contents

1 Prerequisites
2 Install Kerberos client, SSSD and tools
3 Authenticating with Kerberos
4 Configuring realmd
5 Joining The Host to the Active Directory Domain
6 Controlling Who Can Log In
7 Enumerating Users and Groups
8 Automatic Home Directories
9 Internals
10 Links

1 Prerequisites

SSSD and realmd can be found in the Ubuntu repositories, so installation is easy. But a couple of things must be taken care of first.

The first prerequisite is, make sure you are using your Active Directory DNS servers. They will be used to query the addresses of the domain controllers, and when the domain is joined, DNS records (forward and reverse) are added.

The second prerequisite is that your time keeping should closely match the AD domain controller machines (usually within 5 minutes of each other). Use your domain controllers as NTP time sources, or at least use the same time sources for the domain controllers and the Linux hosts to keep their clocks very close to each other.

2 Install Kerberos client, SSSD and tools

Install the Kerberos client, the realm enrollment tool, the System Security Services Daemon, the AD client tool, and Samba tools:

apt-get install krb5-user realmd sssd sssd-tools adcli samba-common-bin

When prompted, type in your AD Kerberos realm. It should generally be your domain name in capital letters (“koo.fi” becomes “KOO.FI”). If your DNS is working properly, that should be all that is needed for the Kerberos client to work alright. Otherwise you may need to add your servers to /etc/krb5.conf.

3 Authenticating with Kerberos

Try getting a Kerberos ticket as domain administrator:

kinit administrator@KOO.FI
klist

The output of klist should look like this:

Ticket cache: FILE:/tmp/krb5cc_0
Default principal: administrator@KOO.FI

Valid starting     Expires            Service principal
06/16/15 16:23:22  06/17/15 02:23:22  krbtgt/KOO.FI@KOO.FI
    renew until 06/17/15 16:23:18

That shows we now have a ticket valid for some hours, meaning the Kerberos authentication is working fine to the domain controller. We can proceed to configuring the realmd realm enrollment tool which will join us to the domain, and later use this ticket to actually execute the join operation.

4 Configuring realmd

Edit /etc/realmd.conf:

[service]
automatic-install = no

[users]
default-home = /home/%D/%U
default-shell = /bin/bash

[koo.fi]
computer-ou = OU=Linux,DC=koo,DC=fi
automatic-id-mapping = yes
fully-qualified-names = no

The automatic-install=no option will disable automatic installation of packages by realmd.

The default-home=/home/%D/%U option will make the home directories of users be of form /home/DOMAIN/USERNAME, eg. /home/koo.fi/administrator. The default-shell is the shell for users.

The computer-ou option tells where the machine account will be added in AD.

The automatic-id-mapping=yes option makes SSSD use automatic id mapping instead of user and group ids stored in POSIX attributes in AD. The SSSD automatic id mapping is intelligent in that it can guarantee the same UNIX uid and gid on different hosts when all the hosts are using SSSD.

The fully-qualified-names=no option will by default remove the domain part from user and group names. It may result in name collisions, but makes things easier for users since they only have to type in their username part and not the domain part every time.

5 Joining The Host to the Active Directory Domain

You can use the "realm discover" command to see if the Active Directory domain can be discovered. It requires avalid Kerberos ticket as a domain administrator.

realm discover koo.fi

Output should look like:

koo.fi
  type: kerberos
  realm-name: KOO.FI
  domain-name: koo.fi
  configured: no
  server-software: active-directory
  client-software: sssd
  required-package: sssd-tools
  required-package: sssd
  required-package: libnss-sss
  required-package: libpam-sss
  required-package: adcli
  required-package: samba-common-bin

We have all the required packages already installed, so let's just join:

realm join koo.fi
realm list

Output looks like:

koo.fi
  type: kerberos
  realm-name: KOO.FI
  domain-name: koo.fi
  configured: kerberos-member
  server-software: active-directory
  client-software: sssd
  required-package: sssd-tools
  required-package: sssd
  required-package: libnss-sss
  required-package: libpam-sss
  required-package: adcli
  required-package: samba-common-bin
  login-formats: %U
  login-policy: allow-realm-logins

After a successful join, you should be able to resolve individual users and groups using getent:

getent passwd administrator
# administrator:*:364000500:364000513:administrator:/home/koo.fi/administrator:/bin/bash
getent group 'Domain Admins'
# domain admins:*:364000512:administrator

If you run into a "Failed to join the domain" error, try the join with user given as an option:

realm join koo.fi --user=administrator

If you run into a "Necessary packages are not installed" error, you may try to install packagekit:

apt-get install packagekit
killall aptd
killall realmd

And then try again. There's a bug in Launchpad about it.

6 Controlling Who Can Log In

Also at this point you should be able to log in with any AD user id by default. You can control who can and who cannot login with

realm deny --all
realm permit administrator
realm permit -g 'Domain Admins'

You can see the changed policy and permitted logins with

realm list

Output:

koo.fi
  type: kerberos
  realm-name: KOO.FI
  domain-name: koo.fi
  configured: kerberos-member
  server-software: active-directory
  client-software: sssd
  required-package: sssd-tools
  required-package: sssd
  required-package: libnss-sss
  required-package: libpam-sss
  required-package: adcli
  required-package: samba-common-bin
  login-formats: %U
  login-policy: allow-permitted-logins
  permitted-logins: administrator
  permitted-groups: Domain Admins

7 Enumerating Users and Groups

The SSSD configuration file /etc/sssd/sssd.conf was generated by the realm join command and will look something like this:

[sssd]
domains = koo.fi
config_file_version = 2
services = nss, pam

[domain/koo.fi]
ad_domain = koo.fi
krb5_realm = KOO.FI
realmd_tags = manages-system joined-with-adcli
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = False
fallback_homedir = /home/%d/%u
access_provider = ad

For performance reasons by default SSSD will not try to enumerate every user and group from AD, but will only query for them when requested. If you want to enable enumeration, add:

enumerate = True

under your domain stanza in sssd.conf (and "restart sssd"). It is quick for a small directory, but may be slow for a large one. It can be handy for debugging, as you can list all users and groups with

"getent passwd" and "getent group".

8 Automatic Home Directories

To create home directories at first login, add a pam_mkhomedir.so line in /etc/pam.d/common-session:

session [default=1]   pam_permit.so
session requisite     pam_deny.so
session required      pam_permit.so
session optional      pam_umask.so
session required      pam_unix.so
session required      pam_mkhomedir.so  skel=/etc/skel umask=0022
session optional      pam_sss.so
session optional      pam_systemd.so

That's basicly it. At this point you should have a fully functioning AD login for selected users on your Ubuntu server.

9 Internals

The "machine account" Kerberos principal is saved into the file /etc/krb5.keytab. You can list its contents with "klist -kt /etc/krb5.keytab":

Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   3 06/16/15 17:35:57 host/server.koo.fi@KOO.FI
   3 06/16/15 17:35:57 host/server.koo.fi@KOO.FI
   3 06/16/15 17:35:57 host/server.koo.fi@KOO.FI
   3 06/16/15 17:35:57 host/server.koo.fi@KOO.FI
   3 06/16/15 17:35:57 host/server.koo.fi@KOO.FI
   3 06/16/15 17:35:57 host/SERVER@KOO.FI
   3 06/16/15 17:35:57 host/SERVER@KOO.FI
   3 06/16/15 17:35:57 host/SERVER@KOO.FI
   3 06/16/15 17:35:57 host/SERVER@KOO.FI
   3 06/16/15 17:35:57 host/SERVER@KOO.FI
   3 06/16/15 17:35:57 SERVER$@KOO.FI
   3 06/16/15 17:35:57 SERVER$@KOO.FI
   3 06/16/15 17:35:57 SERVER$@KOO.FI
   3 06/16/15 17:35:57 SERVER$@KOO.FI
   3 06/16/15 17:35:57 SERVER$@KOO.FI

Here the local server's hostname is "server". SSSD uses this keytab file and the principals in it to periodically refres Kerberos tickets, which are saved in the file * /var/lib/sss/db/ccache_<REALM>*. For example "klist -c /var/lib/sss/db/ccache_KOO.FI":

Ticket cache: FILE:/var/lib/sss/db/ccache_KOO.FI
Default principal: SERVER$@KOO.FI

Valid starting     Expires            Service principal
06/22/15 13:22:52  06/22/15 23:22:52  krbtgt/KOO.FI@KOO.FI
    renew until 06/23/15 13:22:52
06/22/15 13:22:52  06/22/15 23:22:52  ldap/dc1.koo.fi@
    renew until 06/23/15 13:22:52
06/22/15 13:22:52  06/22/15 23:22:52  ldap/dc1.koo.fi@KOO.FI
    renew until 06/23/15 13:22:52

Here we can see three active tickets. These are used to authenticate the host to AD and fetch information from LDAP.

There are also .ldb files under /var/lib/sss/db which you can dump using tdbdump tool from the tdb-tools package if you want to see internal configuration and cached data.

The IP address of the AD domain controller currently used as KDC can be found in /var/lib/sss/pubconf/kdcinfo.<REALM>.

10 Links

VPN between StrongSwan and SonicWall

2015-04-04T18:36:00+03:00

Here's how to create a site-to-site VPN between StrongSwan and SonicWall. This has been tested with Ubuntu 14.04 and StrongSwan 5.1.2, and SonicWall with SonicOS 5.9 at the other end.

Table of Contents

1 Network Topology
2 StrongSwan
- 2.1 Install
- 2.2 Configure
  - 2.2.1 /etc/ipsec.conf:
  - 2.2.2 /etc/ipsec.secrets:
- 2.3 Start IPsec Service
3 SonicWall
4 Check Status

1 Network Topology

In this example, we will route traffic between two networks that are located at different sites. Our example network topology looks like this:

The VPN tunnel will encrypt all traffic between the endpoints. The green networks will be routed to each other through the encrypted tunnel.

2 StrongSwan

2.1 Install

sudo apt-get install strongswan

2.2 Configure

Two files need editing: /etc/ipsec.conf and /etc/ipsec.secrets. The first one is the configuration file, and the second one contains the pre-shared key the endpoints will use to authenticate each other.

2.2.1 /etc/ipsec.conf:

conn sonicwall
  # This server
  left=192.168.1.1
  leftid=192.168.1.1
  # The network behind this server
  leftsourceip=10.0.1.1
  leftsubnet=10.0.1.0/24
  # The remote SonicWall
  right=192.168.2.1
  rightid=192.168.2.1
  # The network behind remote SonicWall
  rightsubnet=10.0.2.0/24
  # Connection parameters
  keyexchange=ikev2
  authby=psk
  ike=aes256-sha1-modp2048
  esp=aes256-sha1-modp2048
  auto=start

We use IKEv2 for key exchange, with AES-256 for encryption and SHA-1 for hashing. A 2048-bit key means Diffie-Hellman group 14. Authentication is done using a pre-shared key. The connection will be started automatically at start.

2.2.2 /etc/ipsec.secrets:

192.168.1.1 192.168.2.1 : PSK "XXXSECRETXXX"

There should be one line per connection in that file. The format, when using pre-shared keys, is this:

local_ip remote_ip : PSK "your password"

Please set your password to a random string.

2.3 Start IPsec Service

To start the service:

sudo service ipsec start

To make it start at boot:

sudo update-rc.d ipsec defaults

To see the status, run:

sudo ipsec statusall

Status of IKE charon daemon (strongSwan 5.1.2, Linux 3.13.0-48-generic, x86_64):
  uptime: 92 seconds, since Apr 04 14:45:29 2015
  malloc: sbrk 2433024, mmap 0, used 351664, free 2081360
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 2
  loaded plugins: charon test-vectors aes rc2 sha1 sha2 md4 md5 random nonce x509
  revocation constraints pkcs1 pkcs7 pkcs8 pkcs12 pem openssl xcbc cmac hmac ctr
  ccm gcm attr kernel-netlink resolve socket-default stroke updown eap-identity
  addrblock
Listening IP addresses:
  192.168.1.1
  10.0.1.1
Connections:
   sonicwall:  192.168.1.1...192.168.2.1  IKEv2
   sonicwall:   local:  [192.168.1.1] uses pre-shared key authentication
   sonicwall:   remote: [192.168.2.1] uses pre-shared key authentication
   sonicwall:   child:  10.0.1.0/24 === 10.0.2.0/24 TUNNEL

However, you need to configure the other endpoint first before you will see an active connection and a security association.

3 SonicWall

Go to VPN > Settings. Add a new VPN connection.

3.1 VPN Connection > General

The "Shared Secret" field should contain the same random string you put into the /etc/ipsec.secrets file.

3.2 VPN Connection > Network

The remote network zone should be a "VPN" zone.

3.3 VPN Connection > Proposals

For the proposal we must match the parameters in /etc/ipsec.conf.

3.4 VPN Connection > Advanced

In the advanced tab I enabled keep-alive.

3.5 Firewall Rules

Also, remember to add firewall rules to allow traffic to flow between networks in SonicWall. Also do the same if you have Iptables in use on the StrongSwan server.

4 Check Status

At this point, the SonicWall should show a green dot on the VPN policy line you created. If not, check the SonicWall log for error messages.

You should also be able to see an active security association on the StrongSwan host with:

sudo ipsec statusall

Status of IKE charon daemon (strongSwan 5.1.2, Linux 3.13.0-48-generic, x86_64):
  uptime: 92 seconds, since Apr 04 14:45:29 2015
  malloc: sbrk 2433024, mmap 0, used 351664, free 2081360
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 2
  loaded plugins: charon test-vectors aes rc2 sha1 sha2 md4 md5 random nonce x509
  revocation constraints pkcs1 pkcs7 pkcs8 pkcs12 pem openssl xcbc cmac hmac ctr
  ccm gcm attr kernel-netlink resolve socket-default stroke updown eap-identity
  addrblock
Listening IP addresses:
  192.168.1.1
  10.0.1.1
Connections:
   sonicwall:  192.168.1.1...192.168.2.1  IKEv2
   sonicwall:   local:  [192.168.1.1] uses pre-shared key authentication
   sonicwall:   remote: [192.168.2.1] uses pre-shared key authentication
   sonicwall:   child:  10.0.1.0/24 === 10.0.2.0/24 TUNNEL
Security Associations (1 up, 0 connecting):
   sonicwall[2]: ESTABLISHED 86 seconds ago, 192.168.1.1[192.168.1.1]...192.168.2.1[192.168.2.1]
   sonicwall[2]: IKEv2 SPIs: XXX XXX*, pre-shared key reauthentication in 2 hours
   sonicwall[2]: IKE proposal: AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
   sonicwall{2}:  INSTALLED, TUNNEL, ESP SPIs: XXX XXX
   sonicwall{2}:  AES_CBC_256/HMAC_SHA1_96, 7224 bytes_i (86 pkts, 1s ago), 7224 bytes_o (86 pkts, 1s ago), rekeying in 44 minutes
   sonicwall{2}:   10.0.1.0/24 === 10.0.2.0/24

You can control your tunnel(s) with these commands:

sudo ipsec down sonicwall
sudo ipsec up sonicwall

See log files /var/log/auth.log and /var/log/syslog for log messages.

OpenStreetMap Nominatim Server for Geocoding

2015-03-19T18:04:00+02:00

Here's how to install the OpenStreetMap Nominatim service on your own server. It can be used to geocode and reverse geocode addresses and map coordinates. You will also get a web interface which loads map tiles from openstreetmap.org while doing geocoding requests using your own server.

I was faced with thousands of geocoding requests per hour, and with such a high volume it is not wise to burden the global OSM servers. Even with aggressive caching it would have potentially been too much.

I will use Ubuntu 14.04 LTS as the platform. Just a basic install with ssh server. We will install Apache to serve http requests. Make sure you have enough disk space and RAM to hold the data and serve it efficiently. I used the Finland extract, which was about a 200 MB download. The resulting database was 26 GB after importing, indexing and adding Wikipedia data. The Wikipedia data probably actually took more disk space than the OSM data. My server has 4 GB RAM, which seems to be enough for this small data set.

Table of Contents

1 Software requirements
2 Kernel tuning
3 PostgreSQL tuning
4 Dedicated user
5 Database users
6 Download and Compile Nominatim
7 Download Data
8 Import Data
9 Database Production Settings
10 Create the Web Site
- 10.1 Apache configuration
- 10.2 Install the Nominatim Web Site
11 Enable OSM Updates
12 Wrap Up
- 12.1 Useful links

1 Software requirements

PostgreSQL with PostGIS extension:

sudo apt-get install \
  postgresql postgis postgresql-contrib \
  postgresql-server-dev-9.3 postgresql-9.3-postgis-2.1 \
  postgresql-doc-9.3 postgis-doc

Apache with PHP5:

apt-get install \
  apache2 php5 php-pear php5-pgsql php5-json php-db

Git and various tools:

apt-get install \
  wget git autoconf-archive build-essential \
  automake gcc proj-bin

Tool for handling OpenStreetMap data:

apt-get install osmosis

Needed libraries:

apt-get install \
  libxml2-dev libgeos-dev libpq-dev libbz2-dev libtool \
  automake libproj-dev libboost-dev libboost-system-dev \
  libboost-filesystem-dev libboost-thread-dev \
  libgeos-c1 libgeos++-dev lua5.2 liblua5.2-dev \
  libprotobuf-c0-dev protobuf-c-compiler

2 Kernel tuning

We must increase kernel shared memory limits. Also we'll reduce swappiness and set the kernel memory overcommit to be a bit more conservative. Make sure you have anough swap space, more than physical memory is a good idea.

These have been estimated using 4 GB RAM.

sysctl -w kernel.shmmax=4404019200
sysctl -w kernel.shmall=1075200
sysctl vm.overcommit_memory=2
sysctl vm.swappiness=10

echo "kernel.shmmax=4404019200" >> /etc/sysctl.conf
echo "kernel.shmall=1075200" >> /etc/sysctl.conf
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf
echo "vm.swappiness=10" >> /etc/sysctl.conf

3 PostgreSQL tuning

The following directives were set in /etc/postgresql/9.3/main/postgresql.conf:

shared_buffers                = 2048MB
work_mem                      = 50MB
maintenance_work_mem          = 1024M
fsync                         = off    # dangerous!
synchronous_commit            = off
full_page_writes              = off    # dangerous!
checkpoint_segments           = 100
checkpoint_timeout            = 10min
checkpoint_completion_target  = 0.9
effective_cache_size          = 2048M

The buffer and memory values depend on your available RAM so you should set sane values accordingly. These are good for 4 GB RAM.

There are two truly dangerous settings here. Setting fsync off will cause PostgreSQL to report successful commits before the disk has confirmed the data has been written successfully. Setting full_page_writes off exposes the database to the possibility of partially updated data in case of power failure. These options are used here only for the duration of the initial import, because setting them so makes the import much faster. Fsync and full page writes must be turned back on after the import to ensure database consistency.

The synchronous_commit also jeopardizes the durability of committed transactions by reporting successful commits to clients asynchronously, meaning that the transaction may not have been fully written to disk. It does not compromize consistency as the two previous directives discussed. It just means that in the case of power failure, some recent transactions that were already reported to the client as having been committed, may in fact be aborted and rolled back. The database will still be in a consistent state. We'll leave it off because it speeds up the queries, and we don't really care about it in this database. We can always rebuild it from scratch. But if you have other important databases in the same database cluster, I would recommend turning it back on as well, since the setting is cluster-wide.

The maintenance_work_mem will also be reduced to a lower value later, after the import.

Restart PostgreSQL to apply changes:

pg_ctlcluster 9.3 main restart

4 Dedicated user

We'll install the software under this user's home directory (user id "nominatim").

useradd -c 'OpenStreetMap Nominatim' -m -s /bin/bash nominatim

5 Database users

We will also make "nominatim" a database user, as well as the already created "www-data" (created by the apache2 package).

Nominatim will be a superuser and www-data will be a regular one.

This must be done as a database administrator. The "postgres" user by default is one (root is not). Change user:

su - postgres

Create database users:

createuser -sdRe nominatim
createuser -SDRe www-data
exit

6 Download and Compile Nominatim

Su to user nominatim:

su - nominatim

Set some environment variables. We'll use these later. These are the download locations for files and updates. Lines 3 and 4 below are for Finland. Customize to your needs.

Also set the BASE_URL to point to your server and install directory. The last forward slash seems to be important.

WIKIPEDIA_ARTICLES="http://www.nominatim.org/data/wikipedia_article.sql.bin"
WIKIPEDIA_REDIRECTS="http://www.nominatim.org/data/wikipedia_redirect.sql.bin"
OSM_LATEST="http://download.geofabrik.de/europe/finland-latest.osm.pbf"
OSM_UPDATES="http://download.geofabrik.de/europe/finland-updates"
BASE_URL="http://maps.example.org/nominatim/"

You can browse available areas at download.geofabrik.de.

6.1 Clone the Repository from Github

We will use the release 2.3 branch:

git clone --recursive \
  https://github.com/twain47/Nominatim.git \
  --branch release_2.3

cd Nominatim

The --recursive option is needed to clone everything, because the repository contains submodules.

6.2 Compile Nominatim

Simply:

./autogen.sh
./configure
make

If there were no errors, we are good. In case of missing libraries, check that you installed all requirements earlier.

6.3 Configure Local Settings

Some required settings should go to ~/Nominatim/settings/local.php:

cat >settings/local.php <<EOF
<?php
// Paths
@define('CONST_Postgresql_Version', '9.3');
@define('CONST_Postgis_Version', '2.1');
@define('CONST_Website_BaseURL', '$BASE_URL');

// Update process
@define('CONST_Replication_Url', '$OSM_UPDATES');
@define('CONST_Replication_MaxInterval', '86400');
@define('CONST_Replication_Update_Interval', '86400');
@define('CONST_Replication_Recheck_Interval', '900');
?>
EOF

The last four definitions are only required if you are going to do incremental updates.

7 Download Data

OpenStreetMap data (about 200 MB for Finland):

wget -O data/latest.osm.pbf $OSM_LATEST

The OSM data is all that is really required. The rest below are optional.

Wikipedia data (about 1.4 GB):

wget -O data/wikipedia_article.sql.bin $WIKIPEDIA_ARTICLES
wget -O data/wikipedia_redirect.sql.bin $WIKIPEDIA_REDIRECTS

Special phrases for country codes and names (very small):

./utils/specialphrases.php --countries >data/specialphrases_countries.sql

Special search phrases (a few megabytes):

./utils/specialphrases.php --wiki-import >data/specialphrases.sql

Next we'll import all this stuff into the database.

8 Import Data

The utils/setup.php will create a new database called "nominatim" and import the given .pbf file into it. This will take a long time depending on your PostgreSQL settings, available memory, disk speed and size of dataset. The full planet can take days to import on modern hardware. My small dataset took a bit over two hours.

./utils/setup.php \
  --osm-file data/latest.osm.pbf \
  --all --osm2pgsql-cache 1024 2>&1 \
  | tee setup.log

The messages will be saved into setup.log in case you need to look at them later.

If you had the Wikipedia data downloaded, the setup should have imported that automatically and told you about it. You can import the special phrases data if you downloaded it earlier using these commands:

psql -d nominatim -f data/specialphrases_countries.sql
psql -d nominatim -f data/specialphrases.sql

9 Database Production Settings

Now that the import is done, it is time to configure the database to settings that are suitable for production use. The following changes were made in /etc/postgresql/9.3/main/postgresql.conf:

maintenance_work_mem          = 128M
fsync                         = on
full_page_writes              = on

You should also set the synchronous_commit directive to on if you have other databases running on this same database cluster. See the PostgreSQL Tuning section earlier in this post.

Apply changes:

pg_ctlcluster 9.3 main restart

10 Create the Web Site

The following commands have to be run as root.

Create a directory to install the site into and set permissions:

mkdir /var/www/html/nominatim
chown nominatim:www-data /var/www/html/nominatim
chmod 755 /var/www/html/nominatim
chmod g+s /var/www/html/nominatim

Ask bots to keep out:

cat >/var/www/html/nominatim/robots.txt <<'EOF'
User-agent: *
Disallow: /
EOF

10.1 Apache configuration

Edit the default site configuration file /etc/apache2/sites-enabled/000-default.conf and make it look like something like below:

<VirtualHost *:80>
  ServerName maps.example.org
  ServerAdmin webmaster@example.org
  DocumentRoot /var/www/html

  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined

  <Directory "/var/www/html/nominatim">
    Options FollowSymLinks MultiViews
    AddType text/html .php
  </Directory>
</VirtualHost>

Apply changes by restarting web server:

apache2ctl restart

10.2 Install the Nominatim Web Site

The installation should be done as the nominatim user:

su - nominatim
cd Nominatim

Run the setup.php with the option to create the web site:

./utils/setup.php --create-website /var/www/html/nominatim

At this point, the site is ready and you can point your browser to the base URL and try it out. It should look something like this:

11 Enable OSM Updates

We will configure a cron job to have the database updated periodically using diffs from Geofabrik.de. As the nominatim user:

rm settings/configuration.txt
./utils/setup.php --osmosis-init

Enable hierarchical updates in the database (these were off during import to speed things up):

./utils/setup.php --create-functions --enable-diff-updates

Run once to get up to date:

./utils/update.php --import-osmosis --no-npi

There seems to be a time check in the import function that prevents another run immediately. Instead it will wait until the interval configured in the "Configure Local Settings" section earlier in this post has passed (default is 1 day).

We will add the command to crontab to be executed every monday at 03:00. Run:

crontab -e

Add a new line to the crontab file looking like this:

00 03 * * mon /home/nominatim/Nominatim/utils/update.php --import-osmosis --no-npi >>/home/nominatim/nominatim-update.log 2>&1

A log file will be created as /home/nominatim/nominatim-update.log.

12 Wrap Up

That's it! You can get JSON or XML data out of the database with http requests. A couple of examples:

Forward geocoding:

http://maps.example.org/nominatim/search.php?format=jsonv2&q=mannerheimintie, helsinki

Reverse geocoding:

http://maps.example.org/nominatim/reverse.php?format=jsonv2&lat=60.1805&lon=24.9434

Check out the Nominatim documentation for details.

This post was adapted from my notes of such an installation process. Let me know in the comments section if I introduced any mistakes. Other comments are also welcome!

12.1 Useful links

Remove Unwanted Files From Git History

2014-12-18T13:06:00+02:00

Here's an example of how to remove every *.pyc file from every commit in Git history. It is adapted from this Git help page.

Rewrite history

Run git filter-branch, forcing (--force) Git to process—but not check out (--index-filter)—the entire history of every branch and tag (--tag-name-filter cat -- --all), removing the specified files ('git rm --cached --ignore-unmatch *.pyc') and any empty commits generated as a result (--prune-empty).

Be careful! This will overwrite your existing tags.

$ git filter-branch --force --index-filter \
'git rm --cached --ignore-unmatch *.pyc' \
--prune-empty --tag-name-filter cat -- --all

Ref 'refs/heads/master' was rewritten
Ref 'refs/remotes/origin/master' was rewritten
WARNING: Ref 'refs/remotes/origin/master' is unchanged

Add a .gitignore entry

This is to prevent you from committing the files again.

$ echo '*.pyc' >>.gitignore
$ git add .gitignore
$ git commit -m 'Ignore .pyc files'

Force-push to remote

If you have a remote repository, the push must be forced so that all remote branches and tags and rewritten.

$ git push origin --force --all
$ git push origin --force --tags

Fix every copy of the repository

You can delete and re-clone every copy of the repository, or rebase, or delete every affected branch and re-create them.

For example to rebase:

$ git fetch --all
$ git rebase <lastest-commit-id>

Adding Git version control to web sites

2014-08-13T16:20:00+03:00

Here's a little script I made to make it easy to put a number of existing web sites residing under /var/www under version control by Git. Regarding security, it is obviously not a good idea to initialize the Git repository directly under the site directory, since from there it might be served by the web server to the outside world. And that is not what I want. Instead what I want is to have the Git working directory directly in the site directory that is served, and the repository somewhere else. That is perfectly possible with Git, just by setting one or two environment variables.

What this script does is actually create an interactive subshell for working with the separated working directory and repository. The environment is initialized automatically, the umask is set correctly, and the prompt is changed to remind the user she is working inside this particular site's environment. You can deploy it many times for multiple sites, customize it for each site, and keep all the repositories under one root directory, each in their own subdirectory. You don't have to remember any of this, just the name of the site or script. You are ready to work with the site once you simply run the script. Pull from a remote, make changes, commit, push, whatever you want. When you are finished, just type 'exit' to get out of the site's environment. You can then run another script to work on another site.

So for example, if you have two sites, called first.example.org and second.example.org, save two copies of the script on your server under /usr/local/bin or $HOME/bin. The name of the first copy could be first.example.org and the name of the second copy could be second.example.org. Edit both files and set the WORKROOT variable to where your sites are situated (eg. /var/www) and REPOROOT to where you want your repositories to go. These paths can be the same in both files. What must be different is the WORKNAME variable, which should be the name of the directory under which the particular site is. The variables should look something like this:

/usr/local/bin/first.example.org:

WORKNAME="first.example.org"
WORKROOT="/var/www"
REPOROOT="/var/git/www/repos"

* /usr/local/bin/second.example.org*:

WORKNAME="second.example.org"
WORKROOT="/var/www"
REPOROOT="/var/git/www/repos"

This configuration means your site roots are:

/var/www/first.example.org
/var/www/second.example.org

Those will be used as the Git working directories. The corresponding repositories will be:

/var/git/www/repos/first.example.org
/var/git/www/repos/second.example.org.

But you don't really have to remember that because all you need to do is run either of the scripts and just use git as you are used to. When you're done, type 'exit' or hit Ctrl+D and you can run another script to get to another site's environment.

Give yourself permission to execute the script files under /usr/local/bin.

Here's what an example first run looks like. First create the repository root directory and set suitable permissions (assuming you belong to the admin group):

$ sudo mkdir /var/git
$ sudo chgrp admin /var/git
$ sudo chmod g+w /var/git

Then run the first script:

$ first.example.org
Entering shell with environment for first.example.org
Git repository is: /var/git/www/repos/first.example.org
fatal: Not a git repository: '/var/git/www/repos/first.example.org'
Initialize repository with 'git init' and commit
Type 'exit' to get out of the environment

As you can see, there is no Git repository yet. That has to be initialized and a first commit made.

$ git init
Initialized empty Git repository in /var/git/www/repos/first.example.org/
$ git add .
$ git commit -m 'First commit'
[master (root-commit) 03de73b] First commit
 1 file changed, 3 insertions(+)
 create mode 100644 index.html
$ exit

As you can see, this is just typical Git usage. You can configure remotes, push and pull, make changes and commit them.

The second time you enter the site's environment by running the script, the welcome message will look a bit different.

$ first.example.org
Entering shell with environment for first.example.org
Git repository is: /var/git/www/repos/first.example.org
On branch master
nothing to commit, working directory clean
* master 03de73b First commit
Type 'exit' to get out of the environment

After running the second.example.org script you can go and initialize the repository for the second site as well. It is just the same as the first one.

Below is the script. Just save it, and customize the WORKROOT and REPOROOT variables how you like them. Then make as many copies of the script as you have sites, and customize the WORKNAME variable to match each site. Add execute permissions.

#!/bin/bash
#
# This script starts a new interactive shell with GIT_DIR and GIT_WORK_TREE
# set to different locations. You can use git normally, but the .git/
# directory contents will be kept in a different location and not under the
# working directory.
#
# You can customize the three variables below to reflect the locations of
# the the repository and the working tree:

WORKNAME="first.example.org"
WORKROOT="/var/www"
REPOROOT="/var/git/www/repos"

# The $WORKNAME is the name of the working directory. It will also be used
# as the name of the directory for the bare repository.
#
# The $WORKROOT is the root directory under which the working directory is
# situated (meaning the checked out files).
#
# The $REPOROOT is the root directory under which the bare repository is
# situated (meaning all the stuff usually under .git/)

# If set, the umask will be set to this. We want to give full privileges
# to the group.
SETUMASK="0002"

# Put the commands you want to execute inside the new shell before going
# into interactive mode between the two EOF:s below. They will be read
# into the variable $RCFILE and will be executed after the shell has been
# started.
read -r -d '' RCFILE <<'EOF'

# First read the regular bashrc file
source $HOME/.bashrc;

# Then change the prompt to reflect the fact we are in a new environment
export PS1="\n\t ($WORKNAME) \u@\h:\n\w\a\\$ ";

# Display status of current repository
git status && git branch -av || echo "Initialize repository with 'git init' and commit"

echo "Type 'exit' to get out of the environment"

EOF

# This is used in the $RCFILE
export WORKNAME

# === Probably no need to touch stuff below this point ===

echo "Entering shell with environment for $WORKNAME"
export GIT_DIR="$REPOROOT/$WORKNAME"
export GIT_WORK_TREE="$WORKROOT/$WORKNAME"
echo "Git repository is: $GIT_DIR"

# Set umask if variable is set
test -z "$SETUMASK" || umask "$SETUMASK"

# Create directories if they do not exist already
test -e "$GIT_DIR" || mkdir -p "$GIT_DIR"
test -e "$GIT_WORK_TREE" || mkdir -p "$GIT_WORK_TREE"
cd "$GIT_WORK_TREE"

# Start a new interactive shell and keep it running until "exit"
if [ -z "$RCFILE" ]; then
  exec bash -i
else
  exec bash --rcfile <(echo "$RCFILE") -i
fi

It is quite straightforward. First the configuration variables are defined, as well as the RCFILE variable which is read from a here document. Next the important Git variables GIT_DIR and GIT_WORK_TREE are exported. After possibly creating directories, another bash is executed in interactived mode, and the RCFILE variable is fed to it as the rc file to be executed at the start of the shell. So in reality the user will be working in a sub-subshell until she exits.

Upsert Methods for PostgreSQL

2014-01-04T15:31:00+02:00

Update: PostgreSQL 9.5 has been released with INSERT .. ON CONFLICT UPDATE support.

PostgreSQL has no "upsert" or "replace" or "insert .. on duplicate key update" or "merge into" construct to conditionally either insert a new row, or if a row with the key already exists, to either update the existing row with the new values, or first delete the old row and then insert a new one. This has been discussed many times in the posgresql developers mailing lists, and plans to implement the SQL standard MERGE operation have been devised. But there is still no such functionality in PostgreSQL as of version 9.3. Here are some examples on how to implement the functionality in different ways, using either a function, a trigger function or a rule. None of them are perfect. There are trade-offs to be made.

To implement similar functionality, there are a couple of ways. One is to first explicitly check if a row already exists, and then, depending on which semantics, merge or replace, you want, issue either INSERT or UPDATE query (for merge), or either INSERT or DELETE/INSERT queries (for replace).

Another way is to first try to INSERT the new value, and then check if an exception is raised. If not, the insert worked. If an exception is raised, the row exists and must be either UPDATEd (for merge) or DELETEd and a new one INSERTed (for replace).

Doing it for one row is fine, but if you need to do it for thousands of rows in one shot, you will run into either your whole transaction aborting after the first duplicate key violation, or the fact that you must issue selects and inserts/updates/deletes according to application logic over the network for each row. And doing each operation in a separate transaction will be slow. Also, having multiple database sessions doing merges on a table simultaneously may result in problems.

For a very good in-depth discussion on the topic, see this article: Why is UPSERT so complicated?

Here's an example table we'll be working on:

CREATE TABLE test (
  id INTEGER PRIMARY KEY,
  value VARCHAR
);

Let's start merging values. The first two methods need application logic every step of the way to work.

Using SELECT

Conceptually it looks like this:

BEGIN;
SELECT id FROM test WHERE id = 1;
-- If found:
UPDATE test SET value = 'a' WHERE id = 1;
-- Else:
INSERTO INTO test (id, value) VALUES (1, 'a');
-- End if.
SELECT id FROM test WHERE id = 2;
-- If found:
UPDATE test SET value = 'b' WHERE id = 2;
-- Else:
INSERTO INTO test (id, value) VALUES (2, 'b');
-- End if.
.
.
.
COMMIT;

Here, the application must check after each SELECT if the row exists, and issue either an UPDATE or an INSERT. That means a lot of traffic between client and server.

Using Savepoints

Conceptually it looks like this:

BEGIN;
SAVEPOINT a;
INSERT INTO test (id, value) VALUES (1, 'a');
-- If exception:
ROLLBACK TO SAVEPOINT a;
UPDATE test SET value = 'a' WHERE id = 1;
-- end exception.
SAVEPOINT b;
INSERT INTO test (id, value) VALUES (2, 'b');
-- If exception:
ROLLBACK TO SAVEPOINT b;
UPDATE test SET value = 'b' WHERE id = 2;
-- end exception.
.
.
.
COMMIT;

Your code will have to do the exception checking after each INSERT and roll back to the previous savepoint. And your application must do the checking every step of the way.

Using a Function

We can move the checking part of the job to the database server, reducing the needed network round-trips between the database server and the application.

This function will first try to insert, and failing that, will update an existing row. You can use this without fearing that your transaction gets aborted due to unique constraint violations on the way.

CREATE FUNCTION merge_test (the_id INTEGER, the_value VARCHAR) RETURNS VOID AS $$
BEGIN
  BEGIN
    INSERT INTO test (id, value) VALUES (the_id, the_value);
  EXCEPTION WHEN unique_violation THEN
    UPDATE test SET value = the_value WHERE id = the_id;
  END;
END;
$$ LANGUAGE plpgsql;

This makes it possible to issue all the queries in one shot to the server, without needing to use application logic between the queries.

Merge some rows:

BEGIN;
SELECT merge_test(1, 'b');
SELECT merge_test(2, 'c');
COMMIT;

Or even simply:

SELECT merge_test(1, 'b'), merge_test(2, 'c');

This works if you are content with the possibility that some other transaction might delete the row you have just merged before your transaction returns and you think you have an existing row in the database. If you know nobody does deletes on the table, you're fine. Another version which loops until a row exists and is up-to-date can work better. This one is adapted straight from the PostgreSQL documentation:

CREATE FUNCTION loop_merge_test (the_id INTEGER, the_value VARCHAR) RETURNS VOID AS $$
BEGIN
  LOOP
    UPDATE test SET value = the_value WHERE id = the_id;
    IF FOUND THEN
      RETURN;
    END IF;
    -- Not found, try insert and check exceptions in case someone inserted
    -- the same key concurrently as we speak
    BEGIN
      INSERT INTO test (id, value) VALUES (the_id, the_value);
      RETURN;
    EXCEPTION WHEN unique_violation THEN
      -- do nothing; just loop back to the UPDATE
    END;
  END LOOP;
END;
$$ LANGUAGE plpgsql;

This one will loop until it succeeds in either operation. Using this you can be a bit more sure that after running the merge a row will exist in the table. But, with this one, if there are other unique constraints on the table, we might be in trouble because we did not check which column generated the unique violation. See the documentation for a discussion on this.

Using a Trigger

If the semantics regarding your particular table are such that insert operations should never fail, but should always work as merge operations instead, you can write a trigger function to do it. Let's make a second table for this test first with:

CREATE TABLE test2 (id INTEGER PRIMARY KEY, value VARCHAR);

Then the trigger:

CREATE OR REPLACE FUNCTION before_insert_test2() RETURNS TRIGGER AS $$
BEGIN
  IF (SELECT COUNT(id) FROM test2 WHERE id = NEW.id) = 1 THEN
    -- found; update, and return null to prevent the insert
    UPDATE test2 SET value = NEW.value WHERE id = NEW.id;
    RETURN NULL;
  END IF;

  -- not found; just return row to insert
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER test2_merge BEFORE INSERT ON test2 FOR EACH ROW EXECUTE PROCEDURE before_insert_test2();

There's a caveat, though. If the id you are inserting already exists, your insert operation will return with an inserted row count of 0. That is kind of both correct and incorrect. Correct when you think of it in insert terms, but incorrect when you think of it in merge terms. Just don't test your success by looking at the row count.

test=# insert into test2 values (1, 'a');
INSERT 0 1
test=# insert into test2 values (1, 'b');
INSERT 0 0

You can also write your merge operation into one insert query:

test=# insert into test2 values (1, 'a'), (2, 'b'), (3, 'c'), (4, 'd'), (5, 'e');
INSERT 0 4

So now one row was updated and four were inserted. And all the dirty stuff is handled server side.

But this approach has the possibility of running into an exception if there are multiple sessions working concurrently with the data. So you must re-try the whole operation if it fails.

Using a Rule

Similar functionality as described above using a trigger can be written using the rule system. Let's make a third test table:

CREATE TABLE test3 (id INTEGER PRIMARY KEY, value VARCHAR);

This rule will test whether a row with the given id exists, and if it does, it will issue an update query instead:

CREATE OR REPLACE RULE test3_merge AS
  ON INSERT TO test3
  WHERE (EXISTS (SELECT 1 FROM test3 WHERE id = NEW.id))
  DO INSTEAD
    UPDATE test3 SET value = NEW.value WHERE id = NEW.id;

This has the same caveat regarding the reported row count as the trigger way. Your row count will be incremented by 0 if the row already existed and was merely updated.

test=# insert into test3 values (1, 'a');
INSERT 0 1
test=# insert into test3 values (1, 'A'), (2, 'B'), (3, 'c');
INSERT 0 2

There is an extra SELECT query here, but the PostgreSQL rule system is clever, so it will actually only issue one INSERT operation and one UPDATE operation on the table for the whole data (use EXPLAIN to see the query plan yourself). Triggers are always fired for every row, so they are slower than rules.

The rule method seems like the fastest way of doing this particular merge. Not by any measured benchmark, but just my personal gut feeling. But this method has the possibility of exceptions in concurrent situations, so you must be ready to handle them in your application code, and re-try when needed.

SQLAlchemy Declarative Class Reflector

2013-12-20T17:40:00+02:00

SQLAlchemy has a nice reflection facility which takes for example a database table name as argument and produces a Table object out of it for manipulation. However, those objects do not behave like the objects produced by declarative classes, which are easier to work with. Here's a little class that helps to bridge that gap by reflecting proper declarative classes from database tables. It has only been tested with PostgreSQL, but it may work with other databases as well.

The class SQLReflector requires the preconfigured SQLAlchemy database engine as argument. Example (try in IPython):

# Make a connection string
import getpass
password = getpass.getpass()
connstr = "postgresql+psycopg2://username:%s@localhost/dbname" % password

# Create an engine
from sqlalchemy import create_engine
engine = create_engine(connstr)

# Create the SQLReflector object using the engine
from sqlreflector import SQLReflector
reflector = SQLReflector(engine)

# Start a session (you can use sqlreflector.session as well)
from sqlalchemy.orm import sessionmaker
sessionmaker = sessionmaker(bind=engine)
session = sessionmaker()

The instantiated SQLReflector object can be used to reflect database objects using the following methods:

reflect_table(schema_name, table_name): reflects a single table into a class
reflect_tables(schema_name, table_name_list): reflects multiple tables at once from one schema
reflect_schema(schema_name): reflects all tables and sequences in the given schema
reflect_database(): reflects everything from a database

So to reflect the table "my_table" in schema "public", write:

MyTable = reflector.reflect_table("public", "my_table")

Now you can query it like any declarative class:

myrow = session.query(MyTable).filter(MyTable.id == 1).one()

The reflect_table method accepts a column_definitions argument. You can use that to define missing stuff, such as primary keys for views, or foreign key relationships not defined in the database.

For example PostgreSQL views don't have primary keys, which causes problems with SQLAlchemy. To define a primary key while reflecting the class, you can write something like this:

from sqlalchemy import Column, Integer
MyView  = reflector.reflect_table(
              'my_schema',
              'my_view',
              column_definitions=
              (
                  Column('id_col', Integer, primary_key=True),
              )
          )

To define an explicit foreign key, you can write something like:

from sqlalchemy import Column, Integer, ForeignKey
MyTable = reflector.reflect_class(
              'my_schema',
              'my_table',
              column_definitions=
              (
                  Column(
                      'fk_col',
                      Integer,
                      ForeignKey('foreign_schema.foreign_table.foreign_column')
                  )
              )
          )

You can find all the tables you have reflected in the SQLReflector instance, under the classes property. Every schema you have used is there, and under each schema every table you have reflected. So if you run reflector.reflect_database(), you should find all you schemas under reflector.classes, and every table like this:

reflector.classes.<schema_name>.<class_name>

But remember, if you have any views defined in a PostgreSQL database, the reflect_database() call will fail, because you must explicitly add a primary key constraint to every view to please SQLAlchemy.

There are a couple of helper methods to snoop around the database and see what's there. The method get_schema_names() will give a list of schema names in the database. The method get_table_names(schema_name) will give a list of table names in a given schema. There is a full sqlalchemy.engine.reflection.Inspector object in the instantiated SQLReflector object (the class member name is, not surprisingly, "inspector"). You can use that to further inspect the database.

SQL objects can have names containing for example accented letters and special characters. Python 2 allows only letters, numbers and underscores in identifier names. Python 3 allows some unicode characters in identifiers. For compatibility's sake, SQLReflector mangles SQL identifiers so that the resulting Python identifiers contain only letters, numbers and underscores.

By default, table names are converted into CamelCase in such a way, that the name is converted into lowercase, and after that, underscore characters in table names are removed, and the character following the underscore character is changed into uppercase. Also first character is uppercased.

For example:

Table called "my_data" becomes a class called "MyData"
Table called "MyData" becomes a class called "Mydata"

If you don't like this behaviour, you can disable it by giving a "camelcase=False" argument to the constructor. In this case, the schema name sanitizer is used for table names.

Schema names are not converted into CamelCase, but are sanitized from unwanted characters.

You can disable all sanitization by giving the class constructor a "sanitize_names=False" argument. In this case the SQL name is used as the class name whatever it is, possibly resulting in exceptions if the class name contains characters Python does not accept.

Here's the code (sqlreflector.py):

# Author Mikko Kortelainen <mikko.kortelainen@techelp.fi>
# Tested (not very thoroughly) with Python 2.7.5 and 3.3.2, and SQLAlchemy 0.8.2

import sqlalchemy
from sqlalchemy import create_engine, MetaData, Table, Sequence
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.engine import reflection

class SQLReflector(object):
    """Reflects SQLAlchemy declarative classes from database.

    Requires the SQLAlchemy database engine as argument. Tested only with
    PostgreSQL. For example:

    from sqlalchemy import create_engine
    connstr = "postgresql+psycopg2://user:pass@host/database?sslmode=require"
    engine = create_engine(connstr)
    reflector = SQLReflector(engine)
    session = reflector.session

    MyTable = reflector.reflect("my_schema", "my_table")
    a_row = session.query(MyTable).filter(MyTable.id == 1).one()

    After a table has been reflected, it is saved in the SQLReflector object.
    You can find them under the classes property:

    reflector.classes.<schemaname>.<tablename>

    By default schema and table names are sanitized so that characters which
    are not letters, numbers or underscores will be stripped from the name.
    Also, initial numeric characters will be stripped. What's left is treated
    as the identifier. For table names (not schema names), the default is
    also to convert the resulting name into CamelCase by first converting
    it to lowercase and after that removing all underscore characters and
    converting the character following the underscore to uppercase. For
    example:

      Table "my_data" becomes class "MyData"
      Table "MyData" becomes class "Mydata"

    You can turn CamelCase conversion off by giving camelcase=False.

    If you give sanitize_names=False, the names are not sanitized at all
    before making Python identifiers out of them. So your schemas and tables
    should have names that are compliant already.
    """

    def __init__(self, database_engine, sanitize_names=True, camelcase=True):
        self.engine = database_engine
        self.sessionmaker = sessionmaker(bind=self.engine)
        self.session = self.sessionmaker()
        self.metadata = MetaData(bind=self.engine)
        self.inspector = reflection.Inspector.from_engine(self.engine)

        self.classes = self.make_empty_object()
        self.sanitize_names = sanitize_names
        self.camelcase = camelcase

    def __repr__(self):
        """Formal representation"""
        return """<SQLReflector sanitize_names=%s camelcase=%s>""" % (
            "True" if self.sanitize_names else "False",
            "True" if self.camelcase else "False")

    def get_schema_names(self):
        """Returns a list of available schemas in database"""
        return self.inspector.get_schema_names()

    def get_table_names(self, schema_name):
        """Returns a list of available tables in a schema.

        For more in-depth exploration you can use the SQLAlchemy reflection.Inspector
        instance inside the class. The member is called "inspector". """
        return self.inspector.get_table_names(schema=schema_name)

    def reflect_table(self, schema_name, table_name, column_definitions=None, \
                            sanitize_names=None, camelcase=None):
        """Reflects a table's metadata and creates an SQLAlchemy declarative
        class with a nice sanitized CamelCase name out of it.

        You may give extra column definitions or pretty much anything you want
        to add to the class in the column_definitions argument. For example to
        define an integer primary key to a PostgreSQL view:

        from sqlalchemy import Column, Integer
        MyView  = reflector.reflect_table(
                      'my_schema',
                      'my_view',
                      column_definitions=
                      (
                          Column('id_col', Integer, primary_key=True),
                      )
                  )

        Or you can define an explicit foreign key if such is not in place in the
        database:

        from sqlalchemy import Column, Integer, ForeignKey
        MyTable = reflector.reflect_class(
                      'my_schema',
                      'my_table',
                      column_definitions=
                      (
                          Column(
                              'fk_col',
                              Integer,
                              ForeignKey('foreign_schema.foreign_table.foreign_column')
                          )
                      )
                  )

        """

        # Sanitize names if requested
        if camelcase is None: camelcase = self.camelcase
        if sanitize_names is None: sanitize_names = self.sanitize_names

        if sanitize_names:
            class_name = self.sanitize_tablename(table_name, camelcase)
            sane_schema_name = self.sanitize_schemaname(schema_name)
        else:
            class_name = table_name
            sane_schema_name = schema_name

        # Make a class from declarative_base()
        Base = declarative_base()
        class_bases = (Base,)

        if column_definitions is not None:
            # Add extra column_definitions argument as part of the new class
            table_definition = Table(
                table_name,
                self.metadata,
                *column_definitions,
                schema=schema_name,
                autoload=True)

        else:
            # No extra definitions
            table_definition = Table(
                table_name,
                self.metadata,
                schema=schema_name,
                autoload=True)

        # Make a proper class out of it
        the_class = type(class_name, class_bases, dict(__table__ = table_definition))

        # Save it as self.classes.<sane_schema_name>.<class_name>
        if not hasattr(self.classes, sane_schema_name):
            setattr(self.classes, sane_schema_name, self.make_empty_object())

        setattr(getattr(self.classes, sane_schema_name), class_name, the_class)

        # Also return it
        return the_class

    def reflect_tables(self, schema_name, table_name_list, \
                             sanitize_names=None, camelcase=None):
        """Makes a list of classes from a list of table names that are all
        defined in the give schema"""
        return [self.reflect_table(schema_name, x, sanitize_names, camelcase) \
                for x in table_name_list]

    def reflect_schema(self, schema_name, sanitize_names=None, camelcase=None):
        """Reflects all tables in a given schema"""
        table_names = self.inspector.get_table_names(schema=schema_name)
        return self.reflect_tables(schema_name, table_names, sanitize_names, camelcase)

    def reflect_database(self, sanitize_names=None, camelcase=None):
        """Reflects all tables in every schema of the database"""
        schema_names = self.inspector.get_schema_names()
        classes = list()
        for schema_name in schema_names:
            classes.extend(self.reflect_schema(schema_name, sanitize_names, camelcase))
        return classes

    def make_empty_object(self):
        """Create an empty object. It is a subclass of object, with the
        distinction, that this one can have its attributes manipulated with
        setattr"""
        EmptyObject = type('EmptyObject', (object,), dict())
        return EmptyObject()

    def underscore_to_camelcase(self, value):
        def camelcase():
            yield str.lower
            while True:
                yield str.capitalize

        c = camelcase()
        return "".join(next(c)(x) if x else '_' for x in value.split("_"))

    def capitalize_first_letter(self, string):
        return string[0].upper() + string[1:]

    def to_unicode(self, thing):
        """For Python 2 and 3 compatibility"""
        import sys
        if sys.version < '3':
            return unicode(thing)
        else:
            return str(thing)

    def to_string(self, thing):
        return str(thing)

    def sanitize_tablename(self, tablename, camelcase=True):
        """Returns a suitable class name derived from tablename
        - accents will be removed from letters
        - other than valid characters will be removed: [^A-Za-z0-9_]
        - underscores will be converted into CamelCase unless camelcase=False
        - initial numeric characters will be stripped
        - first character will be capitalized
        """
        from unicodedata import normalize
        from re import sub

        # Substitute accents and convert to ASCII
        normal = self.to_string(normalize("NFKD", self.to_unicode(tablename)).encode("ASCII", "ignore").decode())

        # Remove any unwanted characters
        new_normal = sub("[^A-Za-z0-9_]+", "", normal)

        # Convert underscore_identifiers to CamelCase
        if camelcase:
            camel = self.underscore_to_camelcase(new_normal)
        else:
            camel = new_normal

        # Remove initial numeric characters (class names cannot start with a
        # number)
        start_alpha = sub("^[0-9]+", "", camel)

        # Capitalize the first letter (only if CamelCase)
        if camelcase:
            classname = self.capitalize_first_letter(start_alpha)
        else:
            classname = start_alpha

        return classname

    def sanitize_schemaname(self, schemaname):
        """This sanitizes a schema name so that it can be used as a Python
        object.
        - accents will be removed from letters
        - other than valid characters will be removed: [^A-Za-z0-9_]
        - initial numeric characters will be stripped (must be A-Za-z_)
        """
        from unicodedata import normalize
        from re import sub

        # Substitute accents and convert to ASCII
        normal = self.to_string(normalize("NFKD", self.to_unicode(schemaname)).encode("ASCII", "ignore").decode())

        # Remove any unwanted characters
        new_normal = sub("[^A-Za-z0-9_]+", "", normal)

        # Remove initial numeric characters (class names cannot start with a
        # number)
        schema_name = sub("^[0-9]+", "", new_normal)

        return schema_name

Editing Tabular Data in Vim

2013-12-20T13:52:00+02:00

Helpful settings for editing tabular data:

setlocal noexpandtab
setlocal nowrap
setlocal tabstop=16 softtabstop=16 shiftwidth=16

Noexpandtab disables expanding of inputted tabs to spaces. Adjust the number of spaces to your liking. Tabstop sets the visual appearance of tab stops (here 16 spaces).

Show tabs visually, disable cursor line:

syntax match Tab /\t/
hi Tab gui=underline guifg=blue ctermbg=blue
setlocal nocursorline

Jump one column left or right Ctrl+H / Ctrl+L in both normal and visual modes to make navigation easier:

nmap <C-L> f<Tab>
nmap <C-H> F<Tab>
vmap <C-L> f<Tab>
vmap <C-H> F<Tab>

Jump 10 lines up or down with Ctrl+K / Ctrl+J:

nmap <C-J> 10j
nmap <C-K> 10k
vmap <C-J> 10j
vmap <C-K> 10k

Disable the behaviour of going to start of line after jumps to make it easier to work on columns:

setlocal nosol

Now you can G or gg and have your cursor in the same column where it was. Makes it easy to select, yank, delete and move columns. Just don't add an empty line at the bottom of the file (you can use "set virtualedit=all" to work around that as well if you want).

Excellent blog post on the subject:

http://acg.github.io/2013/03/29/turn-vim-into-excel-tips-for-tabular-data-editing.html

Developer Mode for the Samsung Galaxy S4

2013-06-13T11:40:00+03:00

From: Developer Mode for the Samsung Galaxy S4

Get to the phone's settings by tapping 'menu' from the home screen and then tapping 'settings'

tap on 'more'
tap on 'software information'
tap on 'About Device'
find the build number, and tap it 7 times
on the 7th tap, a prompt should appear alerting you that Developer Options has been unlocked

You may also wish to:

android update adb
adb kill-server
adb start-server

And after that you must accept the USB connection by tapping OK in the prompt on the device after you connect it.

Set Default search_path for a User in PostgreSQL

2013-04-29T14:03:00+03:00

Here's how to do it:

ALTER ROLE myuser IN DATABASE mydb SET search_path = myschema;

The same syntax works for any configuration parameter, such as localization parameters.

How to prefer IPv4 over IPv6 for some hosts

2013-04-13T20:06:00+03:00

If an IPv6 address is unreachable, but you can reach the IPv4 one, you can set the preference to IPv4 for that particular address by adding a line such as this in /etc/gai.conf:

precedence ::ffff:198.145.11.105/128 100

That will set the preference for host 198.145.11.105 to IPv4.

See gai.conf(5) for details.

Linux IPv6 Router: RADVD + DHCPv6

2013-03-20T21:04:00+02:00

Unlike IPv4, which uses DHCP for configuration, IPv6 uses the Neighbor Discovery Protocol to configure addresses and gateways. Unfortunately, originally the protocol had no means of providing addresses of DNS servers to clients, making it necessary to use DHCPv6 for that purpose. Modern Linux and Mac OS X machines are able to use the IPv6 Router Advertisement Options for DNS Configuration (RFC 6106), but to my knowledge, Windows clients are not able at the moment. Here's how to configure a Linux router using radvd and the ISC DHCP daemon.

Network Configuration and Packet Forwarding

The following configuration has been tested with Ubuntu Server 12.04.2 LTS. For the purposes of this article, the /etc/network/interfaces file looks like this:

# eth0 to Internet
iface eth0 inet6 static
  address 2001:db8:0:1::2
  netmask 64
  gateway 2001:db8:0:1::1

# eth1 to internal network
iface eth1 inet6 static
  address 2001:db8:0:2::1
  netmask 64

Outbound interface is eth0, and inbound interface eth1.

First, to enable IPv6 packet forwarding, put this in your /etc/sysctl.conf:

net.ipv6.conf.all.forwarding=1

And run this to make the change in the running kernel:

sudo sysctl -w net.ipv6.conf.all.forwarding=1

IPv6 Router Advertisement Daemon

Install the router advertisement daemon or radvd with:

sudo apt-get install radvd

Create file /etc/radvd.conf and put your internal interface and prefix there:

interface eth1
{
   AdvSendAdvert on;
   prefix 2001:db8:0:2::/64
   {
   };
};

You can now start the daemon with

sudo service radvd start

That will enable router advertisements on the internal interface. See example configuration files under /usr/share/doc/radvd/examples. See also manual pages for radvd(8) and radvd.conf(5) for more information. The radvdump(8) is a useful tool for watching live router advertisement traffic.

The RDNSS and DNSSL Options

Below is an example radvd.conf which also advertises DNS servers with RDNSS and DNS search path with DNSSL, both of which are specified in RFC 6106. This will work with Linux and Mac OS X clients, but unfortunately Windows does not seem to support it.

interface eth1
{
  AdvSendAdvert on;
  MinRtrAdvInterval 3;
  MaxRtrAdvInterval 10;

  prefix 2001:db8:0:1::/64
  {
  };

  RDNSS 2001:db8:0:1::a 2001:db8:0:1::b
  {
    AdvRDNSSLifetime 10;
  };

  DNSSL koo.fi
  {
    AdvDNSSLLifetime 10;
  };
};

Restart radvd.

DHCPv6

To support Windows, we must install a DHCPv6 compliant DHCP server, such as a recent version of the ISC DHCP daemon:

sudo apt-get install isc-dhcp-server

Create file /etc/dhcp/dhcpd6.conf and put something like this in there:

ddns-update-style none;

default-lease-time 7200;
max-lease-time 86400;

subnet6 2001:db8:0:2::/64 {
  range6
    2001:db8:0:2::1000
    2001:db8:0:2::1fff;

  option dhcp6.name-servers
    2001:db8:0:1::a,
    2001:db8:0:1::b;

  option dhcp6.domain-search
    "koo.fi";
}

We configured a pool of 4096 addresses here (::1000-1fff), plus DNS servers and search path.

Start the dhcpv6 server:

sudo service isc-dhcp-server6 start

If it fails, see /var/log/syslog for error messages. Finally, if everything went ok, add to default runlevels:

sudo update-rc.d isc-dhcp-server6 defaults

Now you should be able to get an IPv6 address and DNS servers with a DHCP client. Additional benefit with DHCP is that you can send more configuration information, such as time server addresses, using DHCP options.

Python 2.7 Windows Installation Checklist

2013-03-12T15:01:00+02:00

Installing Python on Windows is a bit more laborious than on Mac or Linux, because there's a bit of manual configuring to do. Here's an installation checklist to go through to get it done so that you have IPython ready, and you can install packages from PyPI. These instructions are for Python 2.7.

1. Python

Download the Python 2.7 Windows Installer from the Python.org download page. Choose either 32-bit or 64-bit version. Install it. Take note where you install it.

2. PyWin32

Download Python for Windows extensions or "pywin32" here. Install it. It should detect your Python directory. This is optional, but I recommend it.

3. Setuptools

Download setuptools from PyPi

For a 32-bit installation, you can use the setuptools-x.y-win32-py2.7.exe file. Run the installation program, which should detect your installation path.
For a 64-bit installation, download the ez_setup.py and place it in your installation directory.

4. Your PATH

Edit your PATH. Open My Computer properties, from there Advanced Settings, and open the Environment Variables property window. If you have a PATH variable set already, edit that. Otherwise add an environment variable called "PATH" and add your Python installation directory, and also the scripts directory. The paths are separated by semicolons. For example:

C:Python27;C:Python27Scripts;C:Python27ToolsScripts

Here's a screenshot:

If you need other users, or system services to use Python, you can add it to the System Path. If you only use it yourself, add it to your own path.

5. IPython

Open command prompt. Go to the installation folder you chose.

c:
cd \
cd Python27

If you are using the 64-bit version, now is the time to run the ez_setup.py which you downloaded earlier. The defaults are pretty much ok. If you have the 32-bit version, the setuptools should be installed already, so move on.

You can install stuff from the the Python Package Index using the easy_install command, which is in the Scripts folder. If you added it into your path, you can run it from anywhere.

To get a nicer shell, let's install IPython. Run these commands (which were taken from this page):

easy_install distribute
easy_install pyreadline
easy_install ipython

Now you can start the IPython shell by running the ipython command:

C:\Python27>ipython
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]:

Also, to get packages and libraries, just run easy_install to fetch and install it automatically.

If you create your own modules and packages, you can put them in this directory:

C:Python27Libsite-packages

That way you can import them in your scripts easily.

PostgreSQL Database Cluster Migration Notes

2013-03-12T08:38:00+02:00

I had to migrate a pretty big database cluster from one server with PostgreSQL 8.4 to a new server with 9.2 There we multiple apps using the databases, and the downtime had to be minimal. Luckily, the databases were split to multiple schemas, and there were only a few tables in a couple of schemas that were in active use at the time of the migration.

The pg_dump tool was very helpful in this operation. First, I set up ssh public key authentication between the postgres users from the source to the destination server. Then I ran the first full cluster dump to the new empty cluster. This can be done while users are actively using the source database, as long as you know which schemas, tables and sequences are in use, so that they can be copied again after shutting down applications:

postgres@source:~$ pg_dumpall | gzip | ssh destination 'gunzip | psql -e'

The dump is gzipped and sent using ssh to the destination. Watching the run with top showed that gzip and ssh didn't use much CPU compared to what the pg_dump and psql processes took. The -e option of psql tells you which queries exactly it is running.

At this point, there's a snapshot of the source cluster in the destination, with all the metadata, users etc.

Next, after shutting down a particular application which uses a particular database, the following copies just that database:

postgres@source:~$ pg_dump --clean db1 | gzip | ssh destination 'gunzip | psql -e db1'

The --clean option will actually drop all the objects in the destination database and re-create them. Note the destination database name in the psql command.

Another database was so big, that I had to copy individual schemas instead of the whole database, because that would have taken too much time. The following copies just one schema:

postgres@source:~$ pg_dump --clean --schema schema1 db2 | gzip | ssh destination 'gunzip | psql -e db2'

If the whole schema is too big, this one can be used to copy a single table:

postgres@source:~$ pg_dump --clean --table schema2.table1 db2 | gzip | ssh destination 'gunzip | psql -e db2'

The --clean table dump also drops and re-creates the sequences related to the table, so that your sequence numbers are kept intact.

With those commands, it is pretty easy to shut down an application, migrate the data, start up the particular app, and move to migrating the next one. Even if the apps use the same databases and schemas. Just be sure to know which app modifies which database objects. And remember that the schema and table specific dumps will not by default dump blobs.

Sonera CStream Messaging Web Service API with Python and SUDS

2013-03-11T23:51:00+02:00

In this article I will show you how to use the Sonera CStream Messaging Web Service API to send an SMS using Python, and a library called SUDS. The CStream API is two-way service for both sending and receiving messages. You obviously need to pay for the service to get access. After you have your credentials, you can start using the service.

The SUDS is a lightweight SOAP Python client for exploring and using web services. A recent version can be installed on Debian based distros with "sudo apt-get install python-suds", or on almost anything with "pip install suds".

The one thing good to know before starting out, is that the Sonera web service uses WS Security for authentication, which is found in the suds.wsse module.

Let's start hacking in iPython:

kortsi@localhost:~$ ipython
Python 2.7.3 (default, Sep 26 2012, 21:51:14)
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1.rc2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
          -> Python's own help system.
object    -> Details about 'object', use 'object??' for extra details.

In [1]: import logging

In [2]: logging.basicConfig(level=logging.DEBUG)

In [3]: logging.getLogger('suds.client').setLevel(logging.DEBUG)

In [4]: from suds.client import Client

In [5]: client = Client('https://saarni.front.sonera.fi/ws/messaging-v2.wsdl')

At this point you should see a lot of debug messages on the screen. Thats the SUDS querying the WSDL schema and finding out what's possible. You can take a look at it if it you are interested, but there is a better way to explore the metadata.

In [6]: print client

Suds ( https://fedorahosted.org/suds/ )  version: 0.4.1 (beta)  build: R703-20101015

Service ( messaging-v2Service ) tns="http://www.lekab.com/schema/messaging-v2/messages"
   Prefixes (1)
      ns0 = "http://www.lekab.com/schema/messaging-v2/messages"
   Ports (1):
      (messaging-v2Soap11)
         Methods (3):
            GetIncomingMessages(MessageIds messageIds, Attributes attributes, )
            GetMessageStatus(MessageIds messageIds, Attributes attributes, )
            Send(xs:string sender, Recipients recipients, xs:boolean replyable, xs:string conversationId, Priority priority, xs:dateTime scheduledDelivery, xs:dateTime validTo, xs:string tariff, xs:string contentCategory, xs:string contentMetaData, xs:double vat, xs:string referenceId, Data data, Attributes attributes, )
         Types (21):
            Attachment
            Attachments
            Attribute
            Attributes
            Data
            ErrorDetail
            ErrorDetails
            IncomingMessage
            IncomingMessages
            MessageIds
            MessageStatus
            MessageStatuses
            MmsData
            MmsPayload
            OutgoingMessage
            Payload
            Priority
            Recipients
            SmsData
            SmsPayload
            Value

The list of available methods and and data types is what we are interested in.

The three methods there match the API specification. The names are pretty much self-explanatory:

Send() sends text messages
GetMessageStatus() queries the status of sent messages
GetIncomingMessages() retrieves incoming messages

To invoke those methods, we must be able to create data in specified types. To send an SMS, we need to create an SmsData object, wrapped in a Data object. We can have SUDS create them for us:

In [7]: data = client.factory.create('Data')

In [8]: smsdata = client.factory.create('SmsData')

In [9]: data.sms = smsdata

In [10]: print data
(Data){
   sms =
      (SmsData){
         payload =
            (SmsPayload){
               message = None
               udh = None
            }
         encoding = None
         replaceIfPresent = None
         flash = None
         port = None
         attributes =
            (Attributes){
               attribute[] = <empty>
            }
      }
 }

That shows what the Data should look like. Let's also create an SmsPayload object:

In [11]: payload = client.factory.create('SmsPayload')

In [12]: print payload
(SmsPayload){
   message = None
   udh = None
 }

According to the specification, the message attribute must be a "base64 encoded UTF-8 encoded string". Let's test with a string that requires more than plain ASCII:

In [13]: testmessage = u'Tämä on testi! ("This is a test!" in Finnish)'

In [14]: import base64

In [15]: payload.message = base64.encodestring(testmessage.encode('utf8')).strip()

In [16]: print payload
(SmsPayload){
   message = "VMOkbcOkIG9uIHRlc3RpISAoIlRoaXMgaXMgYSB0ZXN0ISIgaW4gRmlubmlzaCk="
   udh = None
 }

In [17]: smsdata.payload = payload

In [18]: print data
(Data){
   sms =
      (SmsData){
         payload =
            (SmsPayload){
               message = "VMOkbcOkIG9uIHRlc3RpISAoIlRoaXMgaXMgYSB0ZXN0ISIgaW4gRmlubmlzaCk="
               udh = None
            }
         encoding = None
         replaceIfPresent = None
         flash = None
         port = None
         attributes =
            (Attributes){
               attribute[] = <empty>
            }
      }
 }

Now we have a payload set. The Data object is now ready for sending. We still need to create a list of recipients:

In [19]: recipients = client.factory.create('Recipients')

In [20]: recipients.recipient.append('358001234567')

In [21]: print recipients
(Recipients){
   recipient[] =
      "358001234567",
 }

The recipient number must be a string, and it must have a country code. A sender id or number is also required, but it can be whatever you choose. We can give it directly to the Send() method when we send the message. But before that, we must set up authentication:

In [22]: from suds.wsse import Security, UsernameToken, Timestamp

In [23]: security = Security()

In [24]: username = UsernameToken('username', 'secretpassword')

In [25]: security.tokens.append(username)

In [26]: timestamp = Timestamp(validity=120)

In [27]: security.tokens.append(timestamp)

In [28]: client.set_options(wsse=security)

We set up a username/password token, and a timestamp. Now we have 120 seconds of time to send a message using the security tokens.

At this point, we should have everything we need to send our message. Use the Send() method:

In [29]: client.service.Send(sender='SENDERID', recipients=recipients, data=data)

You will get a reply from the remote server. It will either tell you that it failed for some reason, but if everything went ok, the message was queued for delivery, and the response object will look something like this:

Out[29]:
(reply){
   messageStatus[] =
      (MessageStatus){
         statusCode = 0
         statusText = "QUEUED"
         id = "1-9876543"
         sender = "SENDERID"
         recipient = "358001234567"
         time = 2013-03-11 15:52:37.000984
         billingStatus = 0
         attributes =
            (Attributes){
               attribute[] =
                  (Attribute){
                     name = "NumberOfMessages"
                     value =
                        (Value){
                           integer = 1
                        }
                  },
                  (Attribute){
                     name = "NumberOfCharacters"
                     value =
                        (Value){
                           integer = 45
                        }
                  },
            }
      },
   attributes =
      (Attributes){
         attribute[] =
            (Attribute){
               name = "TotalNumberOfMessages"
               value =
                  (Value){
                     integer = 1
                  }
            },
      }
 }

The recipient should receive the message shortly.

To wrap it up, let's make a simple class out of it. Put this to a file called cstream.py:

WSDL_URL = 'https://saarni.front.sonera.fi/ws/messaging-v2.wsdl'

from base64 import encodestring as b64enc
from suds.client import Client
from suds.wsse import Security, UsernameToken, Timestamp

class SimpleSmsSender:
  """Sends an SMS using CStream Web Service API

Give your credentials as arguments to the constructor. Example:

  sender = SimpleSmsSender('username', 'secretpassword')
  """

  def __init__(self, username, password):
    self.username = username
    self.password = password
    self.client = Client(WSDL_URL)

  def send(self, sender, recipient, message):
    """Give sender id, recipient number and message as strings. The message
can be a unicode string. Example:

  sender.send('SENDERID', '358001234567', u'Hello world!')
    """
    f = self.client.factory
    data = f.create('Data')
    smsdata = f.create('SmsData')
    data.sms = smsdata
    payload = f.create('SmsPayload')
    payload.message = b64enc(message.encode('utf8')).strip()
    smsdata.payload = payload
    recipients = f.create('Recipients')
    recipients.recipient.append(recipient)
    security = Security()
    username = UsernameToken(self.username, self.password)
    security.tokens.append(username)
    timestamp = Timestamp()
    security.tokens.append(timestamp)
    self.client.set_options(wsse=security)
    return self.client.service.Send(sender=sender,
                                    recipients=recipients,
                                    data=data)

To use:

In [1]: from cstream import SimpleSmsSender

In [2]: sender = SimpleSmsSender('username', 'secretpassword')

In [3]: sender.send('FROM ME', '358001234567', u'Hello world')

That's it! The receiving of messages, the checking of message delivery status, and the handling of exceptions are left as exercises to the reader (and to myself).

Useful links:

Init Script for Daemonizing Non-Forking Processes

2013-03-09T15:18:00+02:00

Sometimes you have an executable which does not fork to the background, but you need to control it with init scripts, so that it does indeed run in the background. Here's a pretty generic init script for that. It allows you to configure these:

DAEMON_NAME="My Little Daemon"
DAEMON_EXECUTABLE="/opt/my_daemon/my_daemon"
DAEMON_OPTIONS=""
DAEMON_HOMEDIR="/opt/my_daemon"
DAEMON_PIDFILE="/var/run/my_daemon.pid"
DAEMON_LOGFILE="/var/log/my_daemon.log"
INIT_SLEEPTIME="2"

The script changes directory to the DAEMON_HOMEDIR, runs DAEMON_EXECUTABLE with DAEMON_OPTIONS and saves the new process id to the DAEMON_PIDFILE file. Remember to use different pidfile for every daemon.

The script also checks whether the process is up and running after INIT_SLEEPTIME seconds. If not, it will fail. It will do the check after stopping as well.

All stdout/stderr output is sent to DAEMON_LOGFILE. If you don't want to log the output, set it to "/dev/null".

You can control init scripts with this part in the beginning of the file:

### BEGIN INIT INFO
# Provides:          my_daemon
# Required-Start:    $network
# Required-Stop:     $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Runs the non-forking program in the background
### END INIT INFO

Ubuntu/Debian update-rc.d will use those defaults to install it. So after you drop the script under /etc/init.d with the name my_daemon, you can install it using this:

sudo update-rc.d my_daemon defaults

To disable temporarily:

sudo update-rc.d my_daemon disable

To re-enable:

sudo update-rc.d my_daemon enable

To remove:

sudo update-rc.d -f my_daemon remove

This script does not change running user id from root to something else, so be careful with it.

To start up the daemon, run:

sudo service my_daemon start

To stop:

sudo service my_daemon stop

Restart:

sudo service my_daemon restart

See status:

sudo service my_daemon status

The Script

Enough explanation. Here's the script:

#!/bin/sh

### BEGIN INIT INFO
# Provides:          my_daemon
# Required-Start:    $network
# Required-Stop:     $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Runs the non-forking program in the background
### END INIT INFO

# Defaults
DAEMON_NAME="My Little Daemon"
DAEMON_EXECUTABLE="/opt/my_daemon/my_daemon"
DAEMON_OPTIONS=""
DAEMON_HOMEDIR="/opt/my_daemon"
DAEMON_PIDFILE="/var/run/my_daemon.pid"
DAEMON_LOGFILE="/var/log/my_daemon.log"
INIT_SLEEPTIME="2"

# Defaults can be overridden in this file
DAEMON_DEFAULTS_FILE="/etc/default/my_daemon"

PATH=/sbin:/bin:/usr/sbin:/usr/bin

# Load alternate configuration if exists
test -f $DAEMON_DEFAULTS_FILE && . $DAEMON_DEFAULTS_FILE

. /lib/lsb/init-functions

# Usually no need to edit below this point. Just have these ready:
#
# * DAEMON_EXECUTABLE - full path to the executable
# * DAEMON_OPTIONS    - options to pass to the executable
# * DAEMON_NAME       - a decriptive name
# * DAEMON_HOMEDIR    - place where to cd before running
# * DAEMON_PIDFILE    - pid file name
# * DAEMON_LOGFILE    - log file name
# * INIT_SLEEPTIME    - how long to wait for startup and shutdown
#
# The rest will be taken care of. Executable is run with "nohup", so no
# need to fork.

is_running () {
  # Test whether pid file exists or not
  test -f $DAEMON_PIDFILE || return 1

  # Test whether process is running or not
  read PID < "$DAEMON_PIDFILE"
  ps -p $PID >/dev/null 2>&1 || return 1

  # Is running
  return 0
}

root_only () {
  if [ "$(id -u)" != "0" ]; then
    echo "Only root should run this operation"
    exit 1
  fi
}

run () {
  if is_running; then
    PID="$(cat $DAEMON_PIDFILE)"
    echo "Daemon is already running as PID $PID"
    return 1
  fi

  cd $DAEMON_HOMEDIR

  nohup $DAEMON_EXECUTABLE $DAEMON_OPTIONS >>$DAEMON_LOGFILE 2>&1 &
  echo $! > $DAEMON_PIDFILE
  read PID < "$DAEMON_PIDFILE"

  sleep $INIT_SLEEPTIME
  if ! is_running; then
    echo "Daemon died immediately after starting. Please check your logs and configurations."
    return 1
  fi

  echo "Daemon is running as PID $PID"
  return 0
}

stop () {
  if is_running; then
    read PID < "$DAEMON_PIDFILE"
    kill $PID
  fi
  sleep $INIT_SLEEPTIME
  if is_running; then
    while is_running; do
      echo "waiting for daemon to die (PID $PID)"
      sleep $INIT_SLEEPTIME
    done
  fi
  rm -f "$DAEMON_PIDFILE"
  return 0
}

case "$1" in
  start)
    root_only
    log_daemon_msg "Starting $DAEMON_NAME"
    run
    log_end_msg $?
    ;;
  stop)
    root_only
    log_daemon_msg "Stopping $DAEMON_NAME"
    stop
    log_end_msg $?
    ;;
  restart)
    root_only
    $0 stop && $0 start
    ;;
  status)
    status_of_proc \
      -p "$DAEMON_PIDFILE" \
      "$DAEMON_EXECUTABLE" \
      "$DAEMON_NAME" \
      && exit 0 \
      || exit $?
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
  ;;
esac

HP ProLiant Management Component Pack on Ubuntu

2013-03-08T12:16:00+02:00

HP seems to have set up a package repository for Ubuntu 12.04, which is an improvement since I last checked a few years ago. To use the repo, add the following line to /etc/apt/sources.list:

deb http://downloads.linux.hp.com/downloads/ManagementComponentPack/ubuntu precise current/non-free

Run "sudo apt-get update".

You can install a number of software packages from the repository:

hpsmh: HP System Management Homepage
hp-smh-template: HP System Management Homepage Templates
cpqacuxe: HP Array Configuration Utility, web-based
hp-snmp-agents: Insight Management SNMP Agents for HP ProLiant Systems
hponcfg: RILOE II/iLO online configuration utility
hp-health: HP System Health Application and Command line Utility Package
hpacucli: HP Command Line Array Configuration Utility
ams: Agentless Monitoring Service for HP ProLiant Gen8 Systems

I installed the iLO configuration utility, System Health App and Array Configuration command line utility.

root@host:/etc/apt# apt-get install hponcfg hp-health hpacucli

I couldn't find a working GPG key so you need to press y or force package installation.

Useful Commands

You can blink the UID light with the hpuid command.

The hpasmcli is a tool to show and set various system parameters.

hpasmcli> show powermeter
Power Meter #1
    Power Reading  : 224

hpasmcli> show powersupply
Power supply #1
    Present  : Yes
    Redundant: No
    Condition: Ok
    Hotplug  : Supported
Power supply #2
    Present  : Yes
    Redundant: No
    Condition: FAILED
    Hotplug  : Supported

A command called "hplog" can be used to view the log:

root@host:~# hplog -v

ID   Severity       Initial Time      Update Time       Count
-------------------------------------------------------------
1026 Repaired       13:44  04/10/2012 13:46  04/10/2012 0001
LOG: System Power Supplies Not Redundant

1027 Repaired       13:46  04/10/2012 13:48  04/10/2012 0001
LOG: System Power Supply: General Failure (Power Supply 2)

1028 Repaired       13:46  04/10/2012 13:48  04/10/2012 0001
LOG: System Power Supplies Not Redundant

1029 Repaired       13:48  04/10/2012 13:49  04/10/2012 0001
LOG: System Power Supply: General Failure (Power Supply 2)

1030 Repaired       13:48  04/10/2012 13:49  04/10/2012 0001
LOG: System Power Supplies Not Redundant

And show system health information (fans, power supplies, temperatures):

root@host:~# hplog -f
ID     TYPE        LOCATION      STATUS  REDUNDANT FAN SPEED
 1  Var. Speed   I/O Zone        Normal     Yes     Medium ( 45)
 2  Var. Speed   I/O Zone        Normal     Yes     Medium ( 45)
 3  Var. Speed   Processor Zone  Normal     Yes     Medium ( 41)
 4  Var. Speed   Processor Zone  Normal     Yes     Low    ( 36)
 5  Var. Speed   Processor Zone  Normal     Yes     Low    ( 36)
 6  Var. Speed   Processor Zone  Normal     Yes     Low    ( 36)

root@host:~# hplog -p
ID     TYPE        LOCATION      STATUS  REDUNDANT
 1  Standard     Pwr. Supply Bay Normal     No
 2  Standard     Pwr. Supply Bay Failed     No

root@host:~# hplog -t
ID     TYPE        LOCATION      STATUS    CURRENT  THRESHOLD
 1  Basic Sensor I/O Zone        Normal   105F/ 41C 158F/ 70C
 2  Basic Sensor Ambient         Normal    68F/ 20C 102F/ 39C
 3  Basic Sensor CPU (1)         Normal    86F/ 30C 260F/127C
 4  Basic Sensor CPU (1)         Normal    86F/ 30C 260F/127C
 5  Basic Sensor Pwr. Supply Bay Normal   111F/ 44C 170F/ 77C
 6  Basic Sensor CPU (2)         Normal    86F/ 30C 260F/127C
 7  Basic Sensor CPU (2)         Normal    86F/ 30C 260F/127C

Array Configuration Utility

The "hpacucli" is a Smart Array configuration tool. Some examples (the prompt is the =>):

root@host:~# hpacucli
HP Array Configuration Utility CLI 9.30.15.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.
=> ctrl all show

Smart Array P400 in Slot 3                (sn: P61630D9SV063I)

=> ctrl slot=3 show

Smart Array P400 in Slot 3
   Bus Interface: PCI
   Slot: 3
   Serial Number: P61630D9SV063I
   Cache Serial Number: PA2270H9VV23RN
   RAID 6 (ADG) Status: Enabled
   Controller Status: OK
   Hardware Revision: D
   Firmware Version: 7.22
   Rebuild Priority: Medium
   Expand Priority: Medium
   Surface Scan Delay: 3 secs
   Surface Scan Mode: Idle
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 15 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 25% Read / 75% Write
   Drive Write Cache: Enabled
   Total Cache Size: 512 MB
   Total Cache Memory Available: 464 MB
   No-Battery Write Cache: Enabled
   Cache Backup Power Source: Batteries
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True

=> ctrl slot=3 array all show

Smart Array P400 in Slot 3

   array A (SAS, Unused Space: 0  MB)

=> ctrl slot=3 array A show

Smart Array P400 in Slot 3

   Array: A
      Interface Type: SAS
      Unused Space: 0  MB
      Status: OK
      Array Type: Data

=> ctrl slot=3 physicaldrive all show

Smart Array P400 in Slot 3

   array A

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 146 GB, OK)

The utility also understands commands directly from the command line:

root@host:~# hpacucli ctrl slot=3 show status

Smart Array P400 in Slot 3
   Controller Status: OK
   Cache Status: OK
   Battery/Capacitor Status: OK

E-mail Alerts

To get e-mail out of the system, I installed Postfix.

root@host:~# apt-get install postfix mailutils

Select "Internet Site". After installation, do a reconfiguration:

root@host:~# dpkg-reconfigure postfix

Select "Internet Site" again. Give your username as the recipient for root and postmaster.

Use the default destination list. No forcing of synchronous updates.

Next question is about where to accept mail from. The default is localhost only, which is good for my purposes, because this is not a proper mail server.

For the rest of the questions I just use defaults. For additional security, you can edit the /etc/postfix/main.cf and change the line

inet_interfaces = all

to:

inet_interfaces = loopback-only

Restart Postfix. To forward all important mail from the system to yourself, edit /etc/aliases:

postmaster: root
root:       your@email.here

Run command

root@host:~# newaliases

Now you will get all root mail. I also like to change root full name, which will show up as the sender of the e-mail. This way I can see which host's root is sending me mail.

chfn -f "Hostname Root" root

Hardware Health Check Script

For Smart Array checking, I wrote this little script and put it in /usr/local/sbin/smart_array_check:

#!/bin/sh

SLOT="3"      # Your controller slot number
ARRAY="A"     # Your array letter
EMAIL="root"  # You can put your e-mail address here

# This function is called if checks don't pass. Send e-mail.
Notify()
{
  SUBJECT=$1
  {
    echo $SUBJECT
    echo
    echo Check date: $(date +"%F %T%:::z")
    echo
    echo Controller Status:
    hpacucli ctrl slot=$SLOT show

    echo Array Status:
    hpacucli ctrl slot=$SLOT array $ARRAY show

    echo Physical Drives:
    hpacucli ctrl slot=$SLOT physicaldrive all show

    echo Physical Drive Details:
    for DRIVE in $(hpacucli ctrl slot=$SLOT physicaldrive all show | grep physicaldrive | awk '{ print $2 }')
    do
      hpacucli ctrl slot=$SLOT physicaldrive $DRIVE show
    done
  } | mail -s "$SUBJECT" $EMAIL
}

# Check that there's a line saying 'Controller Status: OK' etc.

hpacucli ctrl slot=$SLOT show status \
  | grep -q 'Controller Status: OK'  \
  || Notify "Smart Array CONTROLLER FAILURE at $(hostname)"

hpacucli ctrl slot=$SLOT show status \
  | grep -q 'Cache Status: OK'       \
  || Notify "Smart Array CACHE FAILURE at $(hostname)"

hpacucli ctrl slot=$SLOT show status       \
  | grep -q 'Battery/Capacitor Status: OK' \
  || Notify "Smart Array BATTERY FAILURE at $(hostname)"

hpacucli ctrl slot=$SLOT array $ARRAY show \
  | grep -q 'Status: OK' \
  || Notify "Smart Array ARRAY FAILURE at $(hostname)"

# This is for testing:
#Notify "This is a test"

For other hardware health checks I wrote this one and put it in /usr/local/sbin/hw_health_check:

#!/bin/sh

EMAIL="root"  # You can put your e-mail address here

# This function is called if checks don't pass
Notify()
{
  # Something went wrong. Send e-mail
  SUBJECT=$1
  {
    echo $SUBJECT
    echo
    echo Check date: $(date +"%F %T%:::z")
    echo
    echo '== Power Supply Status =='
    echo
    hplog -p

    echo '== System Fan Status =='
    echo
    hplog -f

    echo '== Temperatures =='
    echo
    hplog -t

    echo '== HP System Log =='
    hplog -v

  } | mail -s "$SUBJECT" $EMAIL
}

# Power supply check with hplog -p:
#
# ID     TYPE        LOCATION      STATUS  REDUNDANT
#  1  Standard     Pwr. Supply Bay Normal     No
#  2  Standard     Pwr. Supply Bay Normal     No
#
# A failed power supply looks like this:
#
# ID     TYPE        LOCATION      STATUS  REDUNDANT
#  1  Standard     Pwr. Supply Bay Normal     No
#  2  Standard     Pwr. Supply Bay Failed     No
#
# A removed power supply looks like this:
#
# ID     TYPE        LOCATION      STATUS  REDUNDANT
#  1  Standard     Pwr. Supply Bay Normal     No
#  2  Standard     Pwr. Supply Bay Absent     No
#
# The total number of power supplies should equal the number of
# power supplies with the status "Normal"

TOTAL_PSU_COUNT=$(hplog -p | tail -n +2 | head -n -1 | wc -l)
OK_PSU_COUNT=$(hplog -p | tail -n +2 | head -n -1 | grep -c Normal)

if [ "$TOTAL_PSU_COUNT" != "$OK_PSU_COUNT" ]
then
  Notify "SYSTEM POWER SUPPLY PROBLEM at $(hostname)"
fi

# Fan check with hplog -f:
#
# ID     TYPE        LOCATION      STATUS  REDUNDANT FAN SPEED
#  1  Var. Speed   I/O Zone        Normal     Yes     Medium ( 45)
#  2  Var. Speed   I/O Zone        Normal     Yes     Medium ( 45)
#  3  Var. Speed   Processor Zone  Normal     Yes     Medium ( 41)
#  4  Var. Speed   Processor Zone  Normal     Yes     Low    ( 36)
#  5  Var. Speed   Processor Zone  Normal     Yes     Low    ( 36)
#  6  Var. Speed   Processor Zone  Normal     Yes     Low    ( 36)
#
# The total number of fans should equal the number of
# fans with the status "Normal"

TOTAL_FAN_COUNT=$(hplog -f | tail -n +2 | head -n -1 | wc -l)
OK_FAN_COUNT=$(hplog -f | tail -n +2 | head -n -1 | grep -c Normal)

if [ "$TOTAL_FAN_COUNT" != "$OK_FAN_COUNT" ]
then
  Notify "SYSTEM FAN PROBLEM at $(hostname)"
fi

# Temperature check with hplog -t:
#
# ID     TYPE        LOCATION      STATUS    CURRENT  THRESHOLD
#  1  Basic Sensor I/O Zone        Normal   105F/ 41C 158F/ 70C
#  2  Basic Sensor Ambient         Normal    69F/ 21C 102F/ 39C
#  3  Basic Sensor CPU (1)         Normal    86F/ 30C 260F/127C
#  4  Basic Sensor CPU (1)         Normal    86F/ 30C 260F/127C
#  5  Basic Sensor Pwr. Supply Bay Normal   111F/ 44C 170F/ 77C
#  6  Basic Sensor CPU (2)         Normal    86F/ 30C 260F/127C
#  7  Basic Sensor CPU (2)         Normal    86F/ 30C 260F/127C
#
# The total number of temperature readings should equal the number of
# temperature readings with the status "Normal"

TOTAL_TEMP_COUNT=$(hplog -t | tail -n +2 | head -n -1 | wc -l)
OK_TEMP_COUNT=$(hplog -t | tail -n +2 | head -n -1 | grep -c Normal)

if [ "$TOTAL_TEMP_COUNT" != "$OK_TEMP_COUNT" ]
then
  Notify "SYSTEM TEMPERATURE PROBLEM at $(hostname)"
fi

# This is for testing:
#Notify "This is a test"

Add both scripts to crontab with "crontab -e":

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
0 0,6,18,12 * * * smart_array_check
0 0,6,18,12 * * * hw_health_check

That will run the checks four times a day and e-mail every time there is a failure.

Links:

HP Software Delivery Repository

HP iLO2 Virtual Serial Console on Ubuntu

2013-03-07T23:54:00+02:00

To get a virtual serial console, you need to enable the iLO virtual serial port. I had mine set up like this:

iLO 2 Virtual Serial Port: COM2 0x2F8 IRQ 3

I also enabled ssh access in the iLO web interface. This way I can ssh into the iLO and see all BIOS messages using the "vsp" command. I can even go to the BIOS setup (RBSU) by pressing "ESC-9".

To be able to control GRUB via the virtual serial port, it has to be configured. I changed the following lines in /etc/default/grub:

GRUB_HIDDEN_TIMEOUT=15
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=15
GRUB_CMDLINE_LINUX="text console=tty0 console=ttyS1,115200n8"
GRUB_TERMINAL=console

You must run "update-grub" after editing that file to generate the boot configuration.

The GRUB_HIDDEN_TIMEOUT is the number of seconds to wait for ESC before booting the default kernel. You can press ESC and select another kernel using the virtual serial console.

The GRUB_HIDDEN_TIMEOUT_QUIET=false shows a countdown, so you know when exactly to press ESC.

The GRUB_CMDLINE_LINUX directs Linux kernel messages to both console (tty0) and serial port COM2 (ttyS1). You can configure the speed (115200) in the iLO setup.

To get a login prompt after boot, I added a file named /etc/init/ttyS1.conf with the following contents:

# ttyS1 - getty
#
# This service maintains a getty on tty1 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345] and (
            not-container or
            container CONTAINER=lxc or
            container CONTAINER=lxc-libvirt)

stop on runlevel [!2345]

respawn
exec /sbin/getty -8 115200 ttyS1

That will start a getty on the virtual serial port.

Now I can log in via the iLO if I mess up the network configuration. I can also perform salvage operations at boot time in case filesystem checks fail.

This has been tested on Ubuntu Server 12.04.2 LTS x86_64.

Upgrading HP Proliant iLO2 Firmware with Ubuntu Server

2013-03-07T23:01:00+02:00

I downloaded the firmware from HP site. It was named CP019022.scexe. I tried uploading it in the iLO2 web interface, but it was rejected. Next, I copied it to the server, gave it execute permissions and ran it:

root@host:~# ./CP019022.scexe
./CP019022.scexe: 153: ./CP019022.scexe: pushd: not found
./CP019022.scexe: 158: ./CP019022.scexe: popd: not found
./CP019022.scexe: 96: ./CP019022.scexe: ./flash_ilo2: not found

Not working. This is a fresh installation of Ubuntu Server 12.04.2 LTS x86_64. So I had to start looking closer. The file is a bash script but the hashbang is #!/bin/sh, which puts bash into old Bourne shell mode. No pushd/popd there.Second try with bash proper:

root@host:~# bash CP019022.scexe
CP019022.scexe: line 96: ./flash_ilo2: No such file or directory

Still not working. Looking at the code, after the script comes a gzipped binary blob, which seems to be a tar archive. This is unpacked by the script before the blob. The lines to discard before binary starts are in this variable:

_SKIP=347

Let's decompress it:

root@host:~# tail -n +347 CP019022.scexe | gunzip > cp019022.tar

Then take a look at the contents:

root@host:~# tar tvf cp019022.tar
-rw-rw-rw- root/root    322387 2013-01-18 16:36 CP019022.xml
-rwxr-xr-x root/root     76812 2009-09-01 18:46 flash_ilo2
-rw-r-xr-x root/root   3145728 2013-01-17 23:03 ilo2_215.bin
-rw-r-xr-x root/root      5565 2013-01-18 16:20 README.TXT

There's the "flash_ilo2" tool and the "ilo2_215.bin" file which seems to be the actual firmware. Unpack archive:

root@host:~# mkdir cp019022
root@host:~# tar xvf cp019022.tar -C cp019022
CP019022.xml
flash_ilo2
ilo2_215.bin
README.TXT

The readme file says you could actually take the .bin file and use the web interface to upgrade using that. So that's one possibility.

Trying the flash_ilo2 command:

root@host:~/cp019022# ./flash_ilo2
./flash_ilo2: No such file or directory

Doesn't seem to run. Examine the file a bit further:

root@host:~/cp019022# file flash_ilo2
flash_ilo2: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, stripped

So it is a 32-bit executable. Ubuntu server has no 32-bit ELF loader installed by default. That can be fixed with:

root@host:~/cp019022# sudo apt-get install libc6-i386

And:

root@host:~/cp019022# ./flash_ilo2

FLASH_iLO2 v1.12 for Linux (Aug 31 2009)
Copyright 2009 Hewlett-Packard Development Company, L.P.
Firmware image: ilo2_215.bin
Current iLO 2 firmware version  2.05; Serial number ILOCZC7402XR0

Component XML file: CP019022.xml
CP019022.xml reports firmware version 2.15
This operation will update the firmware on the
iLO 2 in this server with version 2.15.
Continue (y/N)?

It works! So seems like the easy fix is installing 32-bit libc. The original .scexe seems to work as well, although there are a couple of error messages.

Automount Anything over SSH

2013-02-21T12:08:00+02:00

First, make sure you can use public key authentication or similar means to connect to ssh servers without typing in your password all the time.

Install sshfs for mounting remote filesystems over ssh, and afuse for automounting FUSE filesystems (sshfs uses fuse).

sudo apt-get install sshfs afuse

For dead connection detection, add this to your ~/.ssh/config:

Host *
ServerAliveInterval 30

That will ping the remote server every 30 seconds and detect a dead peer. This is needed for the sshfs reconnect option to work properly.

Create a directory where to automount your remote filesystems:

mkdir ~/sshfs

Now ask afuse to automount any server you ask for under that directory:

afuse -o mount_template='sshfs -o reconnect %r:/ %m' \
      -o unmount_template='fusermount -u -z %m' \
      ~/sshfs/

Now, go to the sshfs directory and cd to some server you know you can log on to:

cd sshfs
cd www.example.com

If your authentication to www.example.com works, you should now be able to list the root directory there.

Add that afuse command to somewhere where it will be executed when you log in, like ~/.xsession, and you're done. Or use the gnome-session-properties tool or similar.

This should work for other FUSE filesystems, but I haven't yet tested them.

Web Messaging with RabbitMQ-Web-Stomp and SockJS

2013-02-18T23:17:00+02:00

In this article, I will discuss sending messages from server to web browser using RabbitMQ-Web-Stomp as the backend, and a simple JavaScript library to handle the web browser side. SockJS is used as the browser WebSockets library. This setup enables you to directly push messages in realtime from the RabbitMQ message broker to the web client. This makes it possible to deliver status updates to multiple web clients directly from the message broker. No need for special application server software in the middle.

The reverse is also possible. You can push messages from web browsers directly into the RabbitMQ routing system. This article, though, will discuss the former case, from server to client. The infrastructure required for the latter case is the same, with nothing extra needed.

Table of Contents

1 What's Needed
2 RabbitMQ Installation
3 Web Management Plugin and the rabbitmqadmin script
4 Writing Programs with AMQP
5 Messaging in a Web Browser using RabbitMQ-Web-Stomp
6 A simple example
7 The publisher in Python
8 The Consumer in JavaScript
9 Test It
10 Useful links:

1 What's Needed

RabbitMQ is a piece of server software which implements the AMQP messaging standard. Simply put, it maintains a number of queues for holding incoming messages, and enables other software to publish messages to those queues, and fetch messages from the queues. There are API libraries for all major programming languages, including C, Python, Java, Ruby, PHP and, of course, Erlang, because RabbitMQ itself is written in it.

AMQP is an open standard for messaging middleware. In theory, AMQP products from various vendors should be compatible. AMQP is a binary line protocol, which makes it fast, but unsuitable for web apps.

STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol. The RabbitMQ-Web-Stomp plugin can deliver messages to/from web clients using Stomp.

The Stomp over Web Socket library allows running STOMP streams over web sockets.

SockJS is a WebSocket emulation library which gives a consistent API on all browser platforms, including some older ones which have partial or vendor-specific support. Native web sockets are used if the browser supports them.

2 RabbitMQ Installation

Here's the RabbitMQ download page. The stuff discussed here has been tested with the package version 3.0.2 on Ubuntu, which is not yet in the official repositories.

After installation, you can use the rabbitmqctl command to command the server. It will require root privileges.

3 Web Management Plugin and the rabbitmqadmin script

After installing the package, enable the management plugin:

sudo rabbitmq-plugins enable rabbitmq_management

And restart:

sudo /etc/init.d/rabbitmq-server restart

The management plugin provides a web interface for configuring the server. By default it runs at port 15762 (eg. http://localhost:15672/).

The default username is "guest" and the password is "guest". You should obviously change these. It can be done using the web admin interface.

You can download the rabbitmqadmin Python script from http://localhost:15672/cli/ . You can control RabbitMQ with this script without root privileges, if you provide a username and password either on the command line or in ~/.rabbitmqadmin.conf (run "rabbitmqdamin help configuration" for instructions).

4 Writing Programs with AMQP

Here are some very nice tutorials on how to write programs for RabbitMQ:

http://www.rabbitmq.com/getstarted.html

We will be using the same pattern as example number 5. But we will use a web client as the consumer.

5 Messaging in a Web Browser using RabbitMQ-Web-Stomp

The RabbitMQ-Web-Stomp plugin allows web browsers to do messaging using SockJS. This will enable a web browser to keep a connection with the message broker, and receive messages as soon as they appear in the queue. A callback function will be fired every time upon the receival of a message.

Install the Web Stomp plugin:

sudo rabbitmq-plugins enable rabbitmq_web_stomp

Restart RabbitMQ. You should now get a hello page in port 15674 (note the different port number), under /stomp (http://localhost:15674/stomp).

(If you install the rabbitmq_web_stomp_examples plugin as well, you can find examples at http://localhost:15670/. You can download the stomp.js file used in my example from http://localhost:15670/web-stomp-examples/stomp.js)

6 A simple example

To demonstrate a simple use case, let's write two programs. The first one is a Python program for publishing messages, and the other one is a web app for reading those messages.

7 The publisher in Python

This one will use the Pika AMQP client library (sudo apt-get install python-pika, or pip install pika).

Let's call this app publisher.py:

#!/usr/bin/python
import pika

# Use plain credentials for authentication
mq_creds  = pika.PlainCredentials(
    username = "guest",
    password = "guest")

# Use localhost
mq_params = pika.ConnectionParameters(
    host         = "localhost",
    credentials  = mq_creds,
    virtual_host = "/")

# Anyone subscribing to topic "mymessages" receives our messages
mq_exchange    = "amq.topic"
mq_routing_key = "mymessages"

# This a connection object
mq_conn = pika.BlockingConnection(mq_params)

# This is one channel inside the connection
mq_chan = mq_conn.channel()

import readline
print("Press Ctrl+C to quit.\n")

while True:
  text = raw_input("Enter your message: ")
  print("Sending '" + text + "'")
  mq_chan.basic_publish(
      exchange    = mq_exchange,
      routing_key = mq_routing_key,
      body        = text)

That will simply prompt the user for a line of text, and publish it to the amq.topic exchange of the default virtualhost "/", using routing key mymessages.

Run the script with:

python publisher.py

8 The Consumer in JavaScript

Here's an example app which can read messages using SockJS. Let's call it listener-app.html.

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title>Testing RabbitMQ Web Stomp</title>

    <!-- From: http://cdn.sockjs.org/ -->
    <script src="sockjs-0.3.js"></script>

    <!-- From RabbitMQ-Web-Stomp examples -->
    <script src="stomp.js"></script>

    <!-- Our example app -->
    <script src="listener-app.js"></script>

    <style>
      #output {
        border: 1px solid black;
        min-height: 100px;
      }
    </style>
  </head>
  <body>

    <h1>Waiting for messages</h1>

    <div id="output">
      <!-- incoming messages will be printed here -->
    </div>

    </script>
  </body>
</html>

Here's listener-app.js:

// Use SockJS
Stomp.WebSocketClass = SockJS;

// Connection parameters
var mq_username = "guest",
    mq_password = "guest",
    mq_vhost    = "/",
    mq_url      = 'http://' + window.location.hostname + ':15674/stomp',

    // The queue we will read. The /topic/ queues are temporary
    // queues that will be created when the client connects, and
    // removed when the client disconnects. They will receive
    // all messages published in the "amq.topic" exchange, with the
    // given routing key, in this case "mymessages"
    mq_queue    = "/topic/mymessages";

// This is where we print incomoing messages
var output;

// This will be called upon successful connection
function on_connect() {
  output.innerHTML += 'Connected to RabbitMQ-Web-Stomp<br />';
  console.log(client);
  client.subscribe(mq_queue, on_message);
}

// This will be called upon a connection error
function on_connect_error() {
  output.innerHTML += 'Connection failed!<br />';
}

// This will be called upon arrival of a message
function on_message(m) {
  console.log('message received');
  console.log(m);
  output.innerHTML += m.body + '<br />';
}

// Create a client
var client = Stomp.client(mq_url);

window.onload = function () {
  // Fetch output panel
  output = document.getElementById("output");

  // Connect
  client.connect(
    mq_username,
    mq_password,
    on_connect,
    on_connect_error,
    mq_vhost
  );
}

You will also need the sockjs-0.3.js and stomp.js files. I will include them in this post.

9 Test It

If you now point your browser to http://localhost/rabbitmq-web-stomp-example/listener-app.html, you should see a text saying "Connected to RabbitMQ-Web-Stomp".

Now, write a message in the terminal window running the publish.py Python script. It should appear in the browser window immediately.

You can open multiple browser windows, and they should all get the message. You can also open multiple publisher scripts at once.

At this point, you should see some connections, channels and queues appear in the RabbitMQ web management interface. There should be one of each for every browser window and running Python script.

10 Useful links:

Here's the example, unpack it under you web server's document root:

rabbitmq-web-stomp-example.tar

Command-T - Fast file navigation for VIM

2013-02-13T15:51:00+02:00

Ingenious, fast way to find your files:

Command-T - Fast file navigation for VIM : vim online.

I use the following mappings:

" Set leader key to comma
let mapleader = ","

" Command-T shortcuts
nnoremap <silent> <Leader>t :CommandT<CR>
nnoremap <silent> <Leader>b :CommandTBuffer<CR>

Bring up the finder with ,t or "comma-t". Use the :CommandTFlush command to re-read directory.

Temporarily Disable PostgreSQL Triggers

2013-01-08T11:05:00+02:00

To temporarily disable all triggers in a PostgreSQL session, use this:

SET session_replication_role = replica;

That disables all triggers for the current database session only. Useful for bulk operations, but remember to be careful to keep your database consistent.

To re-enable:

SET session_replication_role = DEFAULT;

Disable a Single Trigger

To disable just a single trigger, use ALTER TABLE:

ALTER TABLE mytable DISABLE TRIGGER mytrigger;

The difference to the previous method is that ALTER TABLE will globally disable the trigger, affecting all database sessions, not just the current one.

To disable all triggers for one table:

ALTER TABLE mytable DISABLE TRIGGER ALL;

To re-enable:

ALTER TABLE mytable ENABLE TRIGGER ALL;

Ubuntu 12.04 Active Directory Authentication

2013-01-06T14:07:00+02:00

Update 2015-06-16:Ubuntu 14.04 Active Directory Authentication

Authenticating Linux users against Active Directory has traditionally been hard. There's a multitude of HOWTOs on how to do it, and every one of them seems to do it a bit differently. This is because environments and goals vary, and there are many ways to achieve a particular goal. I will add my version to the mix. This one fetches users and groups from Active Directory LDAP using a machine account added using the Samba tools, and authenticates users to the Active Directory Key Distribution Center using Kerberos.

I will use the LDAP schema that comes with the Microsoft Services for Unix Tools which has later been renamed to Subsystem for UNIX-based Applications (SUA).

I will configure the server to fetch user account and group data from Active Directory LDAP using the nss_ldap Name Service Switch module. This information will be used in addition to the data found in local /etc/passwd and /etc/group files. I will also configure the Kerberos authentication client using MIT Kerberos libraries and tools, and configure the pam_ldap Pluggable Authentication Module to utilize it. The Samba tools are used to add a machine account for the host, as well as a keytab file which will be used for querying Active Directory LDAP. No need to add special user accounts for querying AD.

Still, the whole process is not simple by any means. Here's an outline:

Install and configure the MIT Kerberos client and test that it works
Install and configure the Samba tools
Use the Samba tools to "join" the machine to the domain, or in other words, to create a machine account for it in the AD LDAP
Add forward and reverse DNS records for the machine
Create the host keytab file
Map a suitable Kerberos principal to the new machine account in AD, and test that it works with the keytab file
Make sure there's a valid ticket for the machine account all the time
Extend AD LDAP schema using the SFU/SUA tools
Install LDAP tools and test connecting to the AD and see the internal structure
Install and configure nss_ldap to fetch user and group information from AD
Install and configure pam_ldap to authenticate to AD
Add a PAM directive to create a home directory for a user logging in the first time if such a directory does not yet exist

Table of Contents

1 Install and configure kerberos
2 Install OpenLDAP tools
3 The Active Directory Schema
4 Install and configure nss_ldap and pam_ldap
5 PAM configuration
6 The End Result
7 Useful links

1 Install and configure kerberos

Install the Kerberos library, related PAM library, and also Samba client which we will just use to create a machine account in the domain, and create a keytab file.

sudo aptitude install libpam-krb5 krb5-user smbclient

Type in your AD Kerberos realm when prompted. It should generally be your domain name in capital letters ("koo.fi" becomes "KOO.FI"). If your DNS is working properly, that should be all that is needed for Kerberos client to work in a basic manner. Otherwise you may need to add your servers to /etc/krb5.conf.

Try getting a ticket:

# kinit Administrator@KOO.FI
Password for Administrator@KOO.FI:
# klist

Ticket cache: FILE:/tmp/krb5cc_0
Default principal: Administrator@KOO.FI

Valid starting     Expires            Service principal
06/09/11 12:01:22  06/09/11 22:01:06  krbtgt/KOO.FI@KOO.FI
    renew until 06/10/11 12:01:22

That shows we now have a valid ticket and the Kerberos authentication is working fine to the domain controller.

Some extra configuration at this point in krb5.conf. The appdefaults/pam configuration section is where pam_krb5 module gets its configuration from (see for example http://linux.die.net/man/5/pam_krb5).

[appdefaults]
pam = {
        realm = KOO.FI
        ticket_lifetime = 1d
        renew_lifetime = 1d
        forwardable = true
        proxiable = false
        retain_after_close = false
        minimum_uid = 2
        try_first_pass = true
        ignore_root = true
}
[libdefaults]
        default_realm = KOO.FI
        default_keytab_name = FILE:/etc/krb5.keytab
        default_tkt_enctypes = rc4-hmac des-cbc-crc des-cbc-md5
        default_tgs_enctypes = rc4-hmac des-cbc-crc des-cbc-md5

[realms]
        KOO.FI = {
                auth_to_local = DEFAULT
        }
[domain_realm]
        .koo.fi = KOO.FI
        koo.fi = KOO.FI

The auth_to_local rule will try to parse the local username from the Kerberos principal name, otherwise it will use the principal name itself (see http://linux.die.net/man/5/krb5.conf).

1.1 Configure Samba

We only need the Samba client because it is a convenient tool to create a machine account in Active Directory. Configure Samba client using /etc/samba/smb.conf:

[global]
  netbios name = UBUNTU
  realm = KOO.FI
  workgroup = KOO
  security = ADS
  kerberos method = system keytab

The "system keytab" kerberos method states that the Kerberos system is used to find the keytab file (see http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html#KERBEROSMETHOD). We defined it in krb5.conf above.

1.2 Join to Domain

Join the machine to the domain. This command creates a machine account in AD:

# sudo net ads join -U Administrator
Enter Administrator's password:
Using short domain name -- KOO
Joined 'UBUNTU' to realm 'koo.fi'
DNS update failed!

If you get the "DNS update failed" error message, you just need to create a host record in DNS manually (both forward and reverse).

After this, you should basicly be able to remove smbclient should you wish to do so. I didn't bother.

1.3 Create a Keytab file

A keytab file should automatically be created by net ads join as /etc/krb5.keytab. This was stated earlier in krb5.conf. You can also create the keytab file manually with the following command (from the smbclient package):

# sudo net ads keytab create

Test the keytab with this command:

# sudo kinit -V -k 'UBUNTU$'
Authenticated to Kerberos v5

That means you are able to authenticate with the keytab file using principal UBUNTU$. If you have error messages, you may turn on debug output in krb5.conf:

[logging]
    default = SYSLOG:DEBUG

The next step is to go to your Active Directory domain controller and use the ktpass.exe tool to create a host principal ("host/ubuntu.koo.fi") and map it to the machine account that was created ("UBUNTU$").

C:\Users\Administrator.KOO>ktpass /princ "host/ubuntu.koo.fi@KOO.FI" /mapuser UBUNTU$@KOO.FI
Targeting domain controller: DC1.koo.fi
Using legacy password setting method
Successfully mapped host/ubuntu.koo.fi to ubuntu$.

Now you should be able to run:

# sudo kinit -V -k 'host/ubuntu.koo.fi@KOO.FI'
Using default cache: /tmp/krb5cc_0
Using principal: host/ubuntu.koo.fi@KOO.FI
Authenticated to Kerberos v5

1.4 Make Sure There's a Valid Machine Account Ticket all the time

Now that we have a working keytab file, we'll have to make sure we have a valid ticket all the time, and that it can be accessed by every user who needs it. Active Directory has a maximum ticket lifetime of 10 hours. The keytab file by itself will not be enough to authenticate, but it can be used to get a ticket from a KDC. We can do it with a cron job ("sudo crontab -e"):

0 * * * * kinit -k ‘host/ubuntu.koo.fi@KOO.FI’ -c /tmp/krb5cc_host; chmod +r /tmp/krb5cc_host

That will run kinit every hour as root, and save the host credentials in the Kerberos credentials cache file /etc/krb5cc_host. This file contains the live tickets, and needs to be refreshed periodically. It also needs to be readable by all users who need to read information from the LDAP directory. That is why we add world-readable permissions to it.

Security-wise, it is important to remember, that anyone who has read permissions to the credentials cache can pose as the principal whose keys are in the file. In our case, it is the AD machine account. You may wish to restrict who has access to the file using groups or acls. You could revoke access from nobody:nogroup like this:

0 * * * * kinit -k ‘host/ubuntu.koo.fi@KOO.FI’ -c /tmp/krb5cc_host; chmod +r /tmp/krb5cc_host; setfacl -m u:nobody:--- /tmp/krb5cc_host; setfacl -m g:nogroup:--- /tmp/krb5cc_host

All that has to be on one line in the crontab. Or you could create a shell script for it. You also need to install the "acl" package to get the setfacl command.

To see what credentials are currently in the file, run:

# klist -c /tmp/krb5cc_host

2 Install OpenLDAP tools

While not strictly necessary, it is helpful at this point to test authentication to AD and fetching stuff from it, usind ldap-utils and SASL module:

# sudo aptitude install ldap-utils libsasl2-modules-gssapi-mit

Edit /etc/ldap/ldap.conf (the OpenLDAP tools configuration file) and put your own details there:

BASE   dc=koo,dc=fi
URI ldap://dc1.koo.fi ldap://dc2.koo.fi

Now, after fetching a ticket with the command "kinit username@REALM" you should be able to run plain "ldapsearch" and see the contents of your whole Active Directory (or up to about a thousand entries) using that Kerberos ticket. This will turn out handy in the next section.

You should also be able to authenticate using the principal in your keytab file at this point. Destroy your ticket by running kdestroy. After fetching a ticket with the keytab file by running kinit -V -k 'host/ubuntu.koo.fi@KOO.FI', you should also be able to ldapsearch without problems.

3 The Active Directory Schema

The Active Directory LDAP Schema must be extended by installing the Microsoft Services For Unix on an Active Directory Domain controller. The schema extension adds some extra attributes to the directory that allow you to specify user's ID number, home directory, shell etc. You can edit them in a new tab which should appear in the Active Directory Users and Computers snap-in. It looks something like this:

The attribute names and objectclasses used in the following discussion have been taken from my setup. They may vary depending on which MS SFU version you have installed at the moment, or which version you installed the first time. It is therefore advisable to take a look at your own directory and take the correct attribute names from there. To do that, go to AD Users and Computers, add UNIX attributes for one user with UNIX attributes set up, and also one such group, and then run ldapsearch to see the attributes:

# ldapsearch "(cn=Mikko K. Kortelainen)"

That whill search by the username and should return a list of attributes saved for the user. My attributes look like this (I removed a lot of stuff from here, only relevant attrs showed):

# Mikko K. Kortelainen, Unix, koo.fi
dn: CN=Mikko K. Kortelainen,OU=Unix,DC=koo,DC=fi
cn: Mikko K. Kortelainen
name: Mikko K. Kortelainen
pwdLastSet: 129520857664062022
uid: mikko
uidNumber: 10385
gidNumber: 10019
loginShell: /bin/bash
name: Mikko K. Kortelainen
msSFU30Password:
unixHomeDirectory: /home/user/mikko
msSFU30PosixMemberOf: CN=UNIX users,OU=Unix,DC=koo,DC=fi

Let's look at the group "UNIX users":

# ldapsearch "(cn=UNIX users)"

dn: CN=UNIX users,OU=Unix,DC=koo,DC=fi
cn: UNIX users
msSFU30PosixMember:: ...
msSFU30PosixMember:: ...
msSFU30PosixMember:: ...
msSFU30PosixMember:: ...

The msSFU30PosixMember attributes contain references to the member user accounts. That will be used to map users to groups.

4 Install and configure nss_ldap and pam_ldap

Now, let's install nss_ldap (the name service switch ldap module) and pam_ldap (the PAM ldap module):

# sudo aptitude install libnss-ldap libpam-ldap

Dpkg will ask questions to do some preconfiguring. This configuration will use the Kerberos host keytab:

LDAP server: ldap://dc1.koo.fi ldap://dc2.koo.fi
Distinguished name of the search base: dc=koo,dc=fi
LDAP version to use: 3
Make local root database admin: no
Does the database require login: no
Password hash: md5

Next, let's configure /etc/ldap.conf some more. This configuration file is for both the pam_ldap and nss_ldap. You will need to substitute your own values here. Let's go through it piece by piece.

First some basic information on how to connect to the directory, and timeouts, option to not to be fussy over SSL/TLS certificates and to ignore referrals. If your directory is partitioned you will have to enable referrals. In that case be sure to have connectivity to all domain controllers, as well as a working DNS. The use_sasl on option will change to using Kerberos through SASL. The sasl_auth_id must be the same which you mapped to your machine account with the ktpass Windows command earlier.

The krb5_ccname tells where to find the Kerberos credentials cache file. The file must be readable by any user you wish to be able to read information from AD. It must also be refreshed periodically. This was configured as a cron job earlier in this article.

use_sasl on
sasl_auth_id host/ubuntu.koo.fi
krb5_ccname FILE:/tmp/krb5cc_host
base dc=koo,dc=fi
uri ldap://dc1.koo.fi ldap://dc2.koo.fi
ldap_version 3
timelimit 30
bind_timelimit 30
tls_checkpeer no
referrals no
bind_policy soft

The nss_base_ attributes tell where to look for particular types of objects in the directory. We will simply give the root object with subtree scope so that the whole directory is checked. You may want to restrict this for performance reasons if you have a huge directory, or for security reasons if you have strict permissions in your directory.

scope sub
nss_base_passwd dc=koo,dc=fi?sub
nss_base_shadow dc=koo,dc=fi?sub
nss_base_group dc=koo,dc=fi?sub

The nss_map_attribute is used to map an ldap attribute to a Linux NSS property. The first argument is the NSS property, and the second argument is the LDAP attribute from which to fetch the value. Be sure to double-check the attribute names that they match the ones in your directory.

nss_map_objectclass posixAccount User
nss_map_objectclass shadowAccount User
nss_map_attribute uid uid
nss_map_attribute uidNumber uidNumber
nss_map_attribute gidNumber gidNumber
nss_map_attribute loginShell loginShell
nss_map_attribute gecos name
nss_map_attribute userPassword msSFU30Password
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_attribute shadowLastChange pwdLastSet
nss_map_objectclass posixGroup Group
nss_map_attribute uniqueMember msSFU30PosixMember
nss_map_attribute cn cn
nss_initgroups_ignoreusers backup,bin,clamav,daemon,dovecot,dspam,games,gnats,irc,landscape,libuuid,list,lp,mail,man,messagebus,mysql,nagios,news,ntp,openamq,postfix,postgres,proxy,rabbitmq,root,sshd,statd,sync,sys,syslog,uucp,www-data

The pam_login_attribute is used as the user name for the PAM authentication process. The pam_password says which kind of a protocol to use for changing user passwords if that is needed. You need to install libpam-ldap to take advantage of it.

pam_login_attribute msSFU30Name
pam_filter objectclass=user
pam_password ad

The last change is needed to the file /etc/nsswitch.conf so that the Name Service Switch knows it should be querying AD:

passwd: compat ldap
group: compat ldap
shadow: compat ldap

Only those three lines need editing.

At this stage, you should be able to list users from Active Directory using "getent passwd", and groups using "getent group". If you can't, add line "debug 1" to /etc/ldap.conf and try re-running "getent passwd". It should give you a pretty verbose log of what's going on. You can set the verbosity up to 10 if you like.

If you want to cache the results, you can install the "unscd" package. That will cache users and groups, so that they are not asked from LDAP every time, which makes things faster. You can tune the settings in /etc/nscd.conf.

5 PAM configuration

Now that the NSS part is working properly, let's configure PAM. It should mostly be done automatically already, just double-check it.

This is what my /etc/pam.d/common-auth looks like:

auth   [success=3 default=ignore]  pam_krb5.so minimum_uid=1000
auth    [success=2 default=ignore]  pam_unix.so nullok_secure try_first_pass
auth    [success=1 default=ignore]  pam_ldap.so use_first_pass
auth    requisite           pam_deny.so
auth    required            pam_permit.so

That will enable Kerberos, LDAP and Unix passwd file authentication.

This is /etc/pam.d/common-account:

account    [success=2 new_authtok_reqd=done default=ignore]    pam_unix.so
account [success=1 default=ignore]  pam_ldap.so
account requisite           pam_deny.so
account required            pam_permit.so
account required            pam_krb5.so minimum_uid=1000

/etc/pam.d/common-password:

password   requisite           pam_krb5.so minimum_uid=1000
password    [success=2 default=ignore]  pam_unix.so obscure use_authtok try_first_pass sha512
password    [success=1 user_unknown=ignore default=die] pam_ldap.so use_authtok try_first_pass
password    requisite           pam_deny.so
password    required            pam_permit.so

/etc/pam.d/common-session:

session [default=1] pam_permit.so
session requisite  pam_deny.so
session required   pam_permit.so
session optional   pam_umask.so
session optional   pam_krb5.so minimum_uid=1000
session required   pam_unix.so
session required   pam_mkhomedir.so skel=/etc/skel/ umask=0077
session optional   pam_ldap.so
session optional   pam_ck_connector.so nox11

The only thing I had to change was to add the pam_mkhomedir.so line, which creates a home directory automatically for any user logging in the first time.

6 The End Result

After all that is in place, you should be able to log in locally, or remotely with ssh, using a user account from AD, authenticating to AD with Kerberos, and get your home directory created automatically for you should it not exist already.

7 Useful links

Checking PostgreSQL Disk Usage

2013-01-06T09:36:00+02:00

This query sums total disk space used by the table including indexes and toasted data for the 20 largest tables:

SELECT nspname || '.' || relname AS "relation",
    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
  FROM pg_class C
  LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
  WHERE nspname NOT IN ('pg_catalog', 'information_schema')
    AND C.relkind <> 'i'
    AND nspname !~ '^pg_toast'
  ORDER BY pg_total_relation_size(C.oid) DESC
  LIMIT 20;

via Disk Usage - PostgreSQL wiki.

Apache HTTP authentication against WordPress password database

2012-12-19T18:24:00+02:00

The stock mod_auth_mysql package in Ubuntu is not able to authenticate against the phpass password hashes stored in the WordPress database.

There seems to be a patch lying around to enable phpass authentication in mod_auth_mysql. Its inclusion in mod_auth_mysql has been requested a long time ago, and again more recently, but for one reason or another it has been declined. Inclusion of the patch into the Debian package has also been requested.

Thanks to Peter Lamberg, there are good instructions around on how to apply the patch and enable it. I've made available a pre-compiled 64-bit package here:

libapache2-mod-auth-mysql_4.3.9-13ubuntu3_amd64.

Below are the instructions to compile one from scratch, and after it follows an example configuration.

Please remember, it is always a good practice to use SSL/TLS protection when sending user authentication information over the Internet.

Compiling a Patched Package in Ubuntu 12.04

Make a working directory:

mkdir mod-auth-mysql-phpass
cd mod-auth-mysql-phpass

Get the dependencies and source code for mod_auth_mysql:

sudo apt-get build-dep mod-auth-mysql
apt-get source mod-auth-mysql

Also install fakeroot for the patching to be successful:

sudo apt-get install fakeroot

Go to the source code:

cd mod-auth-mysql-4.3.9

Check patch list:

cat debian/patches/00list

Add a new patch with the last patch in the list as the base (for me it was number 17):

dpatch-edit-patch patch 018-phpass 017-doc_persistent_conn.dpatch

It should print something like this:

dpatch-edit-patch:

Now launching an interactive shell in your work directory. Edit your files.
When you are done, exit the shell. When you exit the shell, your patch will be
automatically updated based on the changes in your work directory.

Download the patch:

wget http://pelam.fi/published_sources/mod-auth-mysql-phpass/patch.diff

Apply patch, then delete it:

patch < patch.diff
rm patch.diff

Exit dpatch-edit-patch:

exit

It should print something like this:

dpatch-edit-patch: /home/user/mod-auth-mysql-phpass/mod-auth-mysql-4.3.9/debian/patches/018-phpass.dpatch created.

Add the new patch to the end of the patch list:

echo 018-phpass.dpatch >> debian/patches/00list

Build the patched version:

dpkg-buildpackage -b -uc

The package should appear one level up in the directory tree:

cd ..

Installing the Patched .deb

Just install:

sudo dpkg --install libapache2-mod-auth-mysql_4.3.9-13ubuntu3_amd64.deb

And make sure it is not upgraded automatically:

echo "libapache2-mod-auth-mysql hold" | sudo dpkg --set-selections

Configuring the Patched mod_auth_mysql

Enable the module:

sudo a2enmod auth_mysql

Read the documentation:

less /usr/share/doc/libapache2-mod-auth-mysql/DIRECTIVES.gz

Create a directory for protected files:

mkdir /var/www/protected

Configure either using .htaccess file /var/www/protected/.htaccess (you must have "AllowOverride AuthConfig Limit" enabled for this to work), or directly to Apache configuration:

# Disable file-based auth
AuthBasicAuthoritative      Off
AuthUserFile                /dev/null

# Enable MySQL auth
AuthMySQL                   On
AuthType                    Basic
AuthName                    "Unauthorized use prohibited"

# Basic information - fill in your own details here
Auth_MySQL_User             DB_USER
Auth_MySQL_Password         DB_PASSWORD
Auth_MySQL_Host             DB_HOST
Auth_MySQL_DB               DB_NAME
Auth_MySQL_CharacterSet     utf8

# The table and fields to use
Auth_MySQL_Password_Table   wp_users
Auth_MySQL_Username_Field   wp_users.user_login
Auth_MySQL_Password_Field   wp_users.user_pass
Auth_MySQL_Encryption_Types PHPass PHP_MD5 # This is where we need the patch

# Any user found in the table can log in
Require                     valid-user

# Users can log in from anywhere
Order                       allow,deny
Allow                       from all

Replace the DB* values with values of your own (you can use the same values you have in wp-config.php).

Add a test file:

echo "test" > /var/www/protected/test.txt

Restart Apache:

sudo service apache2 restart

Now you should be prompted for username and password when you try to fetch the test file. Also, you should be able to log in with your WordPress username and password but with nothing else.

Protecting WordPress with http Authentication

At first it may sound silly, but you may wish to protect the WordPress installation itself using http authentication. This configuration is useful, if you want each user to only log in once anywhere on your site (inside or outside of WordPress) using http authentication.

To make WordPress recognize http-authenticated users, install the HTTP Authentication plugin to WordPress. Then, enable the plugin. The plugin needs no further configuration. Just protect what you want with Apache directives. You can protect the whole site, or just the wp-login.php file and wp-admin directory to protect logins and administration with http auth.

Serving Python scripts with Apache mod_wsgi, part II - mod_rewrite

2012-12-04T07:29:00+02:00

In part I, we learned how to configure Apache to server any .py file as a web application using mod_wsgi. I promised to tell you more about WebOb and multiprocessing and multithreading, and exception handling. I'll save those topics for later articles. Instead, in this part I will talk about using mod_rewrite - if, why and how to get rid of the .py extension. You will need the test apps from part I to try these out.

Removing the .py extension using mod_rewrite

Many people (including myself) will think that they want to get rid of the .py extension in the URLs. There are valid reasons for this:

The URLs would be more readable. As we all know it is much easier to remember and understand a URL like http://localhost/myapp/myfeature, than something like http://localhost/myapp.py?action=myfeature
The URLs would be more portable. You could, say, re-write your page with C or Haskell, or with whatever you desire, and the users would not notice a thing.

There are also valid reasong speaking against rewriting URLs with mod_rewrite:

It might be confusing for the developer, or the application integrator, or the webadmin taking care of the production server. If you have both a script called myapp.py and a directory myapp, which one is going to be called?
mod_rewrite allows some really nasty tricks in the wrong hands, so for security reasons, it is not a good idea to allow anybody on a shared server to just write their own rules without admin review.

Keeping these things in mind, I'll try to show you how to use mod_rewrite to hide the .py extension. First, enable the rewrite module:

sudo a2enmod rewrite

Then add this to Apache site configuration (or .htaccess in /var/www if you have AllowOverride +FileInfo +Options set for your dir):

<Directory /var/www/>
  ... other options ...
  Options +FollowSymLinks
  RewriteEngine on
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_FILENAME}\.py -f
  RewriteRule ^(.*)$ $1.py [L]
</Directory>

Restart Apache. Now you can browse to http://localhost/python_app/hello or http://localhost/python_app/environ. In the latter, observe that the REQUEST_URI variable has no .py extension, while the SCRIPT_NAME still does:

REQUEST_URI: /python_app/environ
SCRIPT_FILENAME: /var/www/python_app/environ.py
SCRIPT_NAME: /python_app/environ.py

Dissecting the rewrite rules

So what do all those Apache directives do? The first line, Options +FollowSymLinks, is required for rewritten URLs to work, otherwise Apache will deny the requests. The RewriteEngine on directive is needed in order to have any rewriting take place at all. The real magic happens in the last three lines. According to the documentation, the RewriteCond directive "Defines a condition under which rewriting will take place", while the RewriteRule directive "Defines rules for the rewriting engine". All the conditions preceding the rule must evaluate to true in order for the rule to be followed and the URL be rewritten.

RewriteCond %{REQUEST_FILENAME} !-d

This states, that the the requested filename (eg. "hello") must not (!) be an existing directory (-d).

RewriteCond %{REQUEST_FILENAME}\.py -f

This states, that the requested file (eg. "hello") with a .py extension added ("hello.py") must be an existing file (-f).

RewriteRule ^(.*)$ $1.py [L]

If the above two conditions were met, the requested URL (eg. "/python_app/hello"), denoted here with the regular expression "^(.*)$", should be rewritten with a .py extension ("/python_app/hello.py"). The $1 is a backreference to the regular expression. The final [L] says that processing should stop here. It has no meaning if you don't have any other rules. But when you add more rules, mod_rewrite will continue evaluating them if you don't tell it to stop here.

Observing URL dispatch after rewriting

Now, what happens if we create a directory called "hello" in the python_app directory? There's already a "hello.py" file, which we should be able to request without .py extension due to our new rewrite rules. Let's try what happens:

sudo mkdir -p /var/www/python_app/hello

Now, http://localhost/python_app/hello goes to the directory. Was this expected? Well, yes it is. After all, the first rewrite condition explicitly stated that the requested file must not be an existing directory for any rewriting to take place.

If you want to have it the other way around, you need to take the first RewriteCond out and do a bit more configuring. If you look closely, Apache actually redirected you to another url with an additional slash at the end (http://localhost/python_app/hello/). The "culprit" for the extra slash is mod_dir and the DirectorySlash directive, which is on by default. Turning it off and taking out the first RewriteCond will make the url without the slash call the script, and with the slash call the directory. Please note that this might be a security risk, because requesting a directory name without a slash will by default list all files in that directory. So if you have a directory, but no script with the same name, your directory contents can be listed.

Anyway, I think the latter behaviour is harder to use and understand. And I think that it is a good idea to avoid having same names for directories and scripts. Have apache handle the URL dispatch up to your script, and you handle it from there on in your code.

Useful resources:

http://httpd.apache.org/docs/current/mod/mod_rewrite.html

Make WordPress Twenty Eleven theme post pages full wide

2012-12-04T03:06:00+02:00

I wanted to make the indivitual post pages 100% wide, and also the headings a bit bigger. Here's the CSS to do it:

/* Make individual posts full-width */
.singular.page .hentry {padding:1em;}
.singular .entry-header,
.singular .entry-content,
.singular footer.entry-meta,
.singular #comments-title {
 width: 100%;
}
/* Fix the edit button placement */
.singular .entry-meta .edit-link a {
  bottom: auto;
  left: 0px;
  right: auto;
  top: 30px;
}
/* Make headings a bit bigger */
.singular .entry-content h1 {
  font-size: 28px;
  margin-top: 44px;
}
.singular .entry-content h2 {
  font-size: 18px;
  margin-top: 28px;
}

Example H1

Example text 1

Example H2

Example text 2

I used the Add to All plugin to do it. Seems to work fine. Thanks go to:

AVH First Defense Against Spam « WordPress Plugins

2012-12-02T23:03:00+02:00

I added the AVH First Defence Against Spam plugin to my site. Please report to me If you can't post comments:

WordPress › AVH First Defense Against Spam « WordPress Plugins.

Serving Python scripts with Apache mod_wsgi, part I

2012-12-02T22:53:00+02:00

I admit it, I'm a long-time PHP programmer. I got stuck with it for a long time with all my web-based stuff, simply because it is so easy to set up. Well, there is no set-up, it just works. You just add pages to your web server root and give a .php extension to them, and they get requests and give back responses. Nice and simple.

I still use PHP for quick and dirty things (like running shell commands through web interfaces - yes, really, really naughty...) For doing more complex work, I prefer Python. But I miss the PHP-like way of just adding pages or "web applications" with zero-setup. I will examine the possibilities in Python for this kind of behaviour in this article.

Please remember, there are many, some say even too many, Python web frameworks available already that handle all this stuff for you "automagically". You are almost certainly better off with using one of them, if you want to get work done. But they all require some kind of setup work.

Then there's the thing with all kinds of frameworks, that instead of you calling some library, the behaviour of which you understand, to do your work, a framework calls your code. That is all right when things work as expected. But whenever there are glitches, you need to start digging around the framework code to see exactly what's wrong. And if it is a big framework, that could mean a lot of digging around. For that reason, if you're a reasonably seasoned programmer, I think it might not be half bad an idea to create your own minimalistic framework, using existing, good quality libraries the behaviour of which you understand, and which you can easily poke in the Python shell if you think something's not working like it should. Or just to see what's available, and try out the available code.

Also, knowing how stuff works never hurts. This article is also about learning how the Apache http requests are dispatched to Python code through mod_wsgi. I insist on knowing how things work, so I'm doing it the hard way.

Apache configuration

Now let's get on to the subject matter. First, we must install the mod_wsgi module for Apache. This is for Python 2.

sudo aptitude install libapache2-mod-wsgi

There's a module for Python 3 as well in the repos if you wish to use it. I haven't tested this with Python 3 but it should work if all the modules are there.

The following Apache directives will make mod_wsgi handle all requests for files with a .py extension. You can save it for example into file /etc/apache2/mods-enabled/wsgi_script.conf:

<IfModule mod_wsgi.c>

  # Handle all .py files with mod_wsgi
  <FilesMatch ".+\.py$">
    SetHandler wsgi-script
  </FilesMatch>

  # Deny access to compiled binaries
  # You should not serve these to anyone
  <FilesMatch ".+\.py(c|o)$">
    Order Deny,Allow
    Deny from all
  </FilesMatch>

</IfModule>

Note that any compiled .pyc or .pyo files must be denied explicitly. You don't want people downloading your compiled code.

One more thing to configure for Apache: the directory where your scripts are must have the ExecCGI option set (for example in /etc/apache2/sites-enabled/000-default):

<Directory /var/www/>
  Options +ExecCGI
</Directory>

This might be a security issue if you have other executable stuff exposed. Now, after reloading the Apache configuration, of course, we are set to add .py pages.

Pages in mod_wsgi and Python

One thing to note about mod_wsgi is that it will require you to set up a function (or any callable) which handles the requests. By default, it calls the function called "application", with a couple of parameters. Here's a simple example. Let's put it in a new directory /var/www/python_app and call it hello.py:

def application(environment, start_response):
  status = '200 OK'
  response_headers = [('Content-type', 'text/html')]
  start_response(status, response_headers)
  page = """
<html>
<body>
<h1>Hello world!</h1>
<p>This is being served using mod_wsgi</p>
</body>
</html>
"""
  return page

Now, point your browser to the hello.py script (for example http://localhost/python_app/hello.py) and you should see a page.

As you can see, mod_wsgi passes two arguments to the function application. The first contains lots of environment variables from the http request, like parameters, headers, server and client information, etc.

The second argument is a function which should be used to start the response. It takes the http status code and the response headers as arguments. The start_response function must be called in order to produce something else than an internal server error to the client.

The last thing we did here, is construct a simple web page, and return it as the content.

The mod_wsgi environment

Now, let's take a look at the environment argument. It is probably easiest to dump it to YAML, so let's install the Python module:

aptitude install python-yaml

Add the following application into for example /var/www/python_app/environ.py:

import yaml

def application(environment, start_response):
  status = '200 OK'
  response_headers = [('Content-type', 'text/plain')]
  start_response(status, response_headers)
  return yaml.dump(environment)

And go check what the page (http://localhost/python_app/environ.py) shows you. There should be a list of environment variables, both from Apache and mod_wsgi. Something like this:

DOCUMENT_ROOT: /var/www
GATEWAY_INTERFACE: CGI/1.1
HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP_ACCEPT_CHARSET: ISO-8859-1,utf-8;q=0.7,*;q=0.3
HTTP_ACCEPT_ENCODING: gzip,deflate,sdch
HTTP_ACCEPT_LANGUAGE: fi-FI,fi;q=0.8,en-US;q=0.6,en;q=0.4
HTTP_CONNECTION: keep-alive
HTTP_HOST: localhost
HTTP_USER_AGENT: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko)
  Ubuntu/12.10 Chromium/22.0.1229.94 Chrome/22.0.1229.94 Safari/537.4
PATH_INFO: ''
QUERY_STRING: ''
REMOTE_ADDR: 127.0.0.1
REMOTE_PORT: '58959'
REQUEST_METHOD: GET
REQUEST_URI: /python_app/environ.py
SCRIPT_FILENAME: /var/www/python_app/environ.py
SCRIPT_NAME: /python_app/environ.py
SERVER_ADDR: 127.0.0.1
SERVER_ADMIN: webmaster@localhost
SERVER_NAME: localhost
SERVER_PORT: '80'
SERVER_PROTOCOL: HTTP/1.1
SERVER_SIGNATURE: '<address>Apache/2.2.22 (Ubuntu) Server at localhost Port 80</address>

  '
SERVER_SOFTWARE: Apache/2.2.22 (Ubuntu)
mod_wsgi.application_group: 127.0.1.1|/python_app/environ.py
mod_wsgi.callable_object: application
mod_wsgi.enable_sendfile: '0'
mod_wsgi.handler_script: ''
mod_wsgi.input_chunked: '0'
mod_wsgi.listener_host: ''
mod_wsgi.listener_port: '80'
mod_wsgi.process_group: ''
mod_wsgi.queue_start: '1354434559450389'
mod_wsgi.request_handler: wsgi-script
mod_wsgi.script_reloading: '1'
mod_wsgi.version: !!python/tuple [3, 4]
wsgi.errors: !!python/object:mod_wsgi.Log {}
wsgi.file_wrapper: !!python/name:None.file_wrapper ''
wsgi.input: !!python/object:mod_wsgi.Input {}
wsgi.multiprocess: true
wsgi.multithread: false
wsgi.run_once: false
wsgi.url_scheme: http
wsgi.version: !!python/tuple [1, 0]

A lot of info! Let's go throught some of those. The definitive reference can be found in PEP 3333.

DOCUMENT_ROOT: /var/www

This is obviously the Apache document root.

PATH_INFO: ''

This is everything in the URL path part that comes after the script name. Here it is empty, because there's nothing after our script name. But if you go to, say http://localhost/python_app/environ.py/here/is/a/path, PATH_INFO will contain the string '/here/is/a/path'.

QUERY_STRING: ''

These are the URL parameters, meaning the stuff that comes after the question mark sign in the URL. We didn't give any parameters, so it is empty. Try this: http://localhost/python_app/environ.py?a=1&b=2. Now the QUERY_STRING should be 'a=1&b=2'.

REQUEST_METHOD: GET

Well, this is the http method used to request the page. It could be GET, POST, PUT, DELETE, HEAD, or possibly something else.

REQUEST_URI: /python_app/environ.py

This is the complete URL requested from the server. For example for http://localhost/python_app/environ.py/here/is/a/path/?a=1&b=2 the REQUEST_URI will contain the string '/python_app/environ.py?/here/is/a/path/?a=1&b=2'.

SCRIPT_FILENAME: /var/www/python_app/environ.py

This is the full file system path of the script on the server. This is "me" from the application's point of view, in the file system.

SCRIPT_NAME: /python_app/environ.py

This is path of the URL containing the script that was called. This is "me" from the application's point of view, in the requested URL.

mod_wsgi.application_group: 127.0.1.1|/python_app/environ.py

mod_wsgi has a concept of "application groups". This means, that you can configure a bunch of applications to run in the same context, or Python (sub-)interpreter. See the WSGIApplicationGroup configuration directive for more information. For the purposes of this article, we are using the default group given to us by mod_wsgi, which means we have a separate group for each of our scripts.

mod_wsgi.callable_object: application

This is the function (or any callable) mod_wsgi will try to call upon receiving a request. It is configurable through the WSGICallableObject configuration directive.

wsgi.errors: !!python/object:mod_wsgi.Log {}

An output stream (file-like object) to which error output can be written. Stuff written here should go to the error log.

wsgi.file_wrapper: !!python/name:None.file_wrapper ''

You can use this to output any file-like object as the response to the requestor. The second argument is a "block-size suggestion" according to the documentation. I don't know what mod_wsgi does with it. Here's how to use it to return a file as a response:

return environ['wsgi.file_wrapper'](filelike, block_size)

wsgi.input: !!python/object:mod_wsgi.Input {}

An input stream (file-like object) from which the HTTP request body bytes can be read. You can read the raw POST or PUT request content through this.

wsgi.multiprocess: true

This tells us whether we are running in multiprocessing mode or not. As we are, we must expect that multiple instances of the script are running in parallel as separate processes. As they are separate processes, we can use our own local variables without worry at this point. What we do have to worry about is using shared objects outside the process, like files.

wsgi.multithread: false

This tells us whether we are running in multithreaded process or not. If this would be true, we would have to be very careful and use locking or thread-local objects. I encourage you to take care of it anyway, if you expect someone will actually be using your application. It is very likely that that someone will enable multithreading and run into very strange, random errors that are hard to debug. But let's not worry about it at this point, we'll dive into this subject in a later article.

wsgi.run_once: false

If this were true, it would mean the script is loaded and executed from scratch for every request. As this is not true, mod_wsgi will actually only reload the script if it detects the file has changed. So you can do all kinds of initializations outside the application callable that stay permanently for the life time of the process. This usually means that the first request will take a bit more time, as that is when the loading and initialization is done, but all subsequent requests the same process handles are much quicker (just executing the application).

The mod_wsgi directive WSGIImportScript can be used to preload scripts before even the first request comes in.

wsgi.url_scheme: http

This would be "https" for SSL-encrypted connections.

Modules and packages

After coding for awhile, you want to start moving stuff to separate modules and package them up. Usually (like from the command line) you could just add module files in the same directory where you are running your Python program. But unfortunately this does not work with mod_wsgi so easily. Let me show you. Add a module /var/www/python_app/mymodule.py:

my_var = "ABC"

def my_fun():
  return "123"

Then, add an import statement in hello.py:

from mymodule import my_var, my_fun

def application(environment, start_response):
  status = '200 OK'
  response_headers = [('Content-type', 'text/html')]
  start_response(status, response_headers)
  page = """
<html>
<body>
<h1>Hello world!</h1>
<p>This is being served using mod_wsgi</p>
</body>
</html>
"""
  return page

If you try to get the page, you'll end up with a 500 Internal Server Error and an error message in the Apache log file saying something like this:

mod_wsgi (pid=12558): Target WSGI script '/var/www/python_app/hello.py' cannot be loaded as Python module.
mod_wsgi (pid=12558): Exception occurred processing WSGI script '/var/www/python_app/hello.py'.
Traceback (most recent call last):
  File "/var/www/python_app/hello.py", line 1, in <module>
    from mymodule import my_var, my_fun
ImportError: No module named mymodule

So the script directory is not in the list of paths where Python is looking for modules and packages. We could do something like this:

import os.path
directory = os.path.split(__file__)[0]
import site
site.addsitedir(directory)

from mymodule import my_var, my_fun

def application(environment, start_response):
  status = '200 OK'
  response_headers = [('Content-type', 'text/html')]
  start_response(status, response_headers)
  page = """
<html>
<body>
<h1>Hello world!</h1>
<p>This is being served using mod_wsgi</p>
</body>
</html>
"""
  return page

Now that works, but it is ugly, and I don't want to be adding boilerplate code like that to every script of mine. There's a mod_wsgi directive called WSGIPythonPath which can be used to achieve the same goal. But it has another problem: you can't include it in any .htaccess file, because it needs to be set outside the scope of any VirtualHost or Directory directive in the Apache configuration. But you can do it if you want. Just add a line like this to your Apache default site configuration (just add it out of the VirtualHost directive - perhaps as the first line in the file, for example):

WSGIPythonPath /var/www/python_app

And reload Apache. Now the import statement will work. The downside is, now all your scripts will use that path as an import dir, which may or may not be what you want. You could also do this:

WSGIPythonPath /var/www

Reload, and make your app directory a Python package:

touch /var/www/python_app/__init__.py

Now you can import your module thus:

from python_app.mymodule import my_var, my_fun

It works, but feels a bit odd as you're actually inside the package doing the import. And web site directories are something that I would like to be able to rename without changing code (common modules should anyway be located somewhere else, like /usr/local/lib/python2.7/site-packages).

So, there are a couple of ways of doing the imports, neither of them perfect. The choice is yours. If you happen to know a nice, clean way of doing imports in a mod_wsgi script, please report to me! Thanks.

Hiding things we don't want to serve

What's left in this section is checking out what happens when the user tries to request our modules, compiled modules, and packages, which we don't want to serve.

First the module: http://localhost/python_app/mymodule.py

That gives a 404 Not Found. Seems fine to me.

Then the compiled module: http://localhost/python_app/mymodule.pyc

That gives a 403 Forbidden. Remember, we explicitly forbade the users from fetching our .pyc or .pyo files in the Apache configuration (right at the beginning of this article).

What about the directory: http://localhost/python_app/

That gives a 403 Forbidden as well. You could add an "index.html" file there to get a default page, but we can do better.

Lastly, what about the package __init__.py file: http://localhost/python_app/__init__.py

That gives a 404 Not Found.

If this mix of 403's and 404's hurts your sense of harmony like it does mine, you can alter the /etc/apache2/mods-enabled/wsgi_script.conf to look like this:

<IfModule mod_wsgi.c>

  # Handle all .py files with mod_wsgi
  <FilesMatch ".+\.py$">
    SetHandler wsgi-script
  </FilesMatch>

  # Deny access to compiled binaries
  RedirectMatch 404 ".+\.py(c|o)$"

  # Add a default Python index page
  DirectoryIndex index.py

</IfModule>

And Apache reload. Now there's a 404 Not Found for all the forbidden stuff. So now you can put your Python applications and helper modules all in the web site document root. If a .py file has the application callable, it is a web page or application, and will be served to the client. If it doesn't have the application callable, it is a page not found.

Also, with the DirectoryIndex directive, you can add a default page for any directory by adding an index.py page (with the application callable).

Now almost everything seems perfect. Let's move on to the last subject for tonight.

WebOb - WSGI request and response objects

Parsing the raw WSGI environment variables for each request and adding the response headers and status codes with robust exception handling is a tedious, error-prone and most of all boring job after awhile. Because I don't want you to start hacking away implementing that next, I will just quickly introduce you to an excellent library for doing all the boring stuff, and a bit more. That library is called WebOb. So let's just:

sudo apt-get install python-webob

And then make a WebOb enabled app (let's make it hello_webob.py):

def application(environment, start_response):
  from webob import Request, Response

  request = Request(environment)
  params = request.params
  post = request.POST

  page = """
<html>
<body>
<h1>Hello world!</h1>
<p>This is being served using mod_wsgi and WebOb</p>
<p>
 <form action="" method="POST">
 a: <input name="a" value="%s" /> <br />
 b: <input name="b" value="%s" /> <br />
 <input type="submit" value="Send" />
 </form
</p>
<p>Your a parameters are: %s</p>
<p>Your b parameters are: %s</p>
</body>
</html>
""" % (post.get('a', ''),
       post.get('b', ''),
       ", ".join(params.getall('a')),
       ", ".join(params.getall('b')))

  response = Response(body = page,
                      content_type = "text/html",
                      charset = "utf8",
                      status = "200 OK")

  return response(environment, start_response)

It's a bit longer than the previous hello.py, but it does a lot more. Try giving it parameters using the inputs, using URL vars, or even both. And it takes extra path arguments, everything at the same time.

This sunday night is getting late so that's all for now. I will write a follow-up on this. I need to discuss WebOb some more, multiprocessing and multithreading issues, exception handling, and there's lots of other stuff to talk about.

Have fun with trying it out! And report any security issues my setup might have, or any other insights.

Edit: Check outpart II - mod_rewrite

Useful links:

mod_wsgi: http://code.google.com/p/modwsgi/
PEP 3333: Python Web Server Gateway Interface v1.0.1: http://www.python.org/dev/peps/pep-3333/
WebOb: http://webob.org/

AWStats Multi-Site Setup

2012-12-01T16:57:00+02:00

Here's how to install and configure AWStats to compile separate statistics about multiple Apache sites or virtual hosts running on a single server. This can be done when you configure each Apache site or virtual host to use a different access log file.

The following instructions have been tested on Ubuntu 12.04.

Install

sudo apt-get install awstats

Configure AWStats for the default site

Edit the /etc/awstats/awstats.conf.local file and not the master configuration file. The main awstats.conf file is a good reference for all the configuration options, of which there are plenty.

The only thing you really need to start gathering stats for your default-site, is enter your web server's default site domain name /etc/awstats/awstats.conf.local:

SiteDomain="my.domain"

That is all that is needed for AWStats to make statistics out of the default site. We will add more configuration files later, one for each site or virtual host we want to get statistics for. But we must fix a couple of things first.

Fix log file permissions

AWStats update script will run every 10 minutes as the user "www-data" (see /etc/cron.d/awstats). The Apache log files are by default not readable by the user www-data, so something must be done about that.

Edit /etc/logrotate.d/apache2 and give Apache log file read permissions to everybody. This may not be a good idea on a server with untrusted users with shell access, but mine has none.

Change the following line:

create 640 root adm

To:

create 644 root adm

Save, and then change the permissions of the existing log files:

sudo chmod o+r /var/log/apache2/*.log

Also, give permission to access the directory:

sudo chmod o+x /var/log/apache2

Ensure logrotate does not skew your statistics

AWStats runs every ten minutes, but logrotate might rotate a log file whenever it gets full, which will cause AWStats to miss some hits.

To ensure AWStats update script is always run before a logfile is rotated, we need to add a script for that. Add the following directory:

sudo mkdir -p /etc/logrotate.d/httpd-prerotate

Add the update script:

sudo tee /etc/logrotate.d/httpd-prerotate/awstats << 'EOF'
#!/bin/sh
if [ -x /usr/share/awstats/tools/update.sh ]; then
  su -l -c /usr/share/awstats/tools/update.sh www-data
fi
EOF

Give execute permission:

sudo chmod ug+x /etc/logrotate.d/httpd-prerotate/awstats

At this point, you can run the statistics collector from the command line:

sudo su -l -c /usr/share/awstats/tools/update.sh www-data

If there are no error messages, you should now have up-to-date statistics collected. Let's get a site up and running to display the stats.

Configure Apache

Now that we have some real data, let's set up a location where we can actually take a look at it. There's an example Apache configuration file in /usr/share/doc/awstats/examples/apache.conf. But don't copy it directly to Apache configuration directory. I added something like this to my default site file under /etc/apache2/sites-enabled:

<Directory /var/lib/awstats>
  Options None
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>

<Directory /usr/share/awstats/icon>
  Options None
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>

<Directory /usr/share/java/awstats>
  Options FollowSymLinks
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>

<Directory /usr/lib/cgi-bin>
  AuthType Basic
  AuthName "Unauthorized use prohibited"
  AuthBasicProvider file
  AuthUserFile /etc/apache2/htpasswd/htpasswd.awstats
  Require valid-user
</Directory>

ScriptAlias /awstats.pl /usr/lib/cgi-bin/awstats.pl
Alias /awstats-icon/ /usr/share/awstats/icon/
Alias /awstatsclasses/ /usr/share/java/awstats/

That will require a username and password to access the stats. It is a good idea to use an SSL-secured version of the default site, and use that instead of unsecured one. You can run "a2ensite default-ssl" to enable the default SSL site.

The above configuration will actually from now on require a password for all your cgi scripts dwelling under /usr/lib/cgi-bin, so if you have something there you wish not to protect, move the awstats.pl to somewhere else and secure that instead.

To add the username and password for access:

sudo mkdir -p /etc/apache2/htpasswd
sudo htpasswd -c /etc/apache2/htpasswd/htpasswd.awstats myusername

Reload Apache configuration:

/etc/init.d/apache2 reload

And then point your browser to http(s)://my.domain/awstats.pl to see some statistics!

Next, let's move on to configuring separate statistics for multiple sites.

Configure AWStats for multiple sites

Place all additional configs in /etc/awstats/. Name the new configs "awstats." + domain.name + ".conf", eg. awstats.my.other.domain.conf.

In each file, include the main awstats.conf file. The first line should be:

Include "/etc/awstats/awstats.conf"

After that, we need to configure at least the SiteDomain directive, the location of the Apache log file to scan, and place to put the statistics data. For example:

Include "/etc/awstats/awstats.conf"
SiteDomain="my.other.domain"
HostAliases="my.other.domain www.my.other.domain"
DirData="/var/lib/awstats/my.other.domain"
LogFile="/var/log/apache2/my.other.domain_access_log"

Make sure you get the LogFile path right (take a look at your Apache configuration to find out where it should be).

Also, create the directory for the statistics:

sudo mkdir -p /var/lib/awstats/my.other.domain
sudo chown www-data:www-data /var/lib/awstats/my.other.domain

The next time AWStats runs, within the next 10 minutes, it will collect statistics about this other site. If you are impatient, or want to see if you made any mistakes with your configuration, you can run it manually:

sudo su -l -c /usr/share/awstats/tools/update.sh www-data

Now we have data for this site.

Repeat the procedure for all other sites you have:

Add a configuration file
Create a stats directory with the right permissions
Gather stats

Accessing statistics for other sites than the default-site

To see the statistics, browse to your default site's awstats.pl script, but give it a "config" url parameter:

http(s)://my.domain/awstats.pl?config=my.other.domain

You can also configure AWStats separately for each site's Apache configuration file, so you can use URLs like this: http(s)://my.other.domain/awstats.pl.

Hunajapurkki: A honeypot link widget for WordPress

2012-12-01T01:06:00+02:00

As a first excercise in creating WordPress plugins, I decided to write an extremely simple widget which can be used to lure spammers' address harvester bots to honeypots, for example ones set up using the Project Honey Pot web site. If you want to help fight spam, just sign up for free at their page and follow the instructions. You can add your honeypot to your WordPress site using the Hunajapurkki widget.You can use either the QuickLink URL, which is extremely simple, or set up your own honeypot on your own server if you're feeling a bit more adventurous. In both cases, you should end up with a URL which is the honeypot. You will add this URL to the Hunajapurkki widget.

Installation of Hunajapurkki

Download Hunajapurkki.zip
Unzip Hunajapurkki.zip to the /wp-content/plugins/ directory.
Go to Appearance -> Widgets and add a Hunajapurkki wherever you wish and as many times as you wish.
For each added widget, you must provide a honeypot URL

Remember, the widget should in no way be visible on the page to anyone browsing the site. There should be nothing, not even empty space, shown where you added the widget.

If you want to confirm the installation was successful, you can look at your page source code and search for "hunajapurkki". Your honeypot link should show up there. In fact, if you look at the source code of my blog's front page, or any page in it, you should be able to find one or two honeypots there.

Technical details

What will be added in place of the widget is html like this:

<div style='display: none;'>
 <a href='http://link-to-the-honeypot'>
   <span style='display: none;'>
     <!-- http://link-to-the-honeypot -->
   </span>
 </a>
</div>

Any modern browser should display nothing on the screen, because of the "display:none" CSS attribute. The a tag intentionally does not have the "display:none" attribute in case the spambot is looking for it. Also, the link has content in it for the same reason.

I have tested with Firefox, Chrome and Opera, and Internet Explorer down to IE7 using

the sidebar of the default WP 3.4 theme, and it seems to behave fine.

In case you are wondering about the strange name of the widget, Hunajapurkki, it comes from the Finnish language, meaning literally 'a honey pot'.

ADT requires org.eclipse.wst.sse.core 0.0.0 but it could not be found

2012-11-29T16:35:00+02:00

In Eclipse 3.8, installing the Android Development tools may fail with the following error:

Cannot complete the install because one or more required items could not be found. Software being installed: Android Development Tools 0.9.9.v201009221407-60953 (com.android.ide.eclipse.adt.feature.group 0.9.9.v201009221407-60953) Missing requirement: Android Development Tools 0.9.9.v201009221407-60953 (com.android.ide.eclipse.adt.feature.group 0.9.9.v201009221407-60953) requires 'org.eclipse.wst.sse.core 0.0.0' but it could not be found

The fix is to add the "Helios" update site in addition to the ADT one. The correct URL is:

http://download.eclipse.org/releases/helios

Thanks go to:

ACPI Shutdown on Virtual Ubuntu

2012-11-29T15:10:00+02:00

If your virtual machine does not shut down when asked by the hypervisor, install the package acpi-support.

sudo apt-get install acpi-support

Works at least with KVM, but I see no reason why it would not work as well in other virtualization platforms, if they just send the ACPI shutdown signal to VM.

Thanks:

Virtualbox ACPI Shutdown on Ubuntu Lucid Lynx « The Redscreen Blog .

Wordpress Logins and Administration over SSL

2012-11-27T09:52:00+02:00

To force SSL logins and the administration interface, edit wp-config.php and add these before the "/* That's all, stop editing! Happy blogging. */" line:

define('FORCE_SSL_LOGIN', true);
define('FORCE_SSL_ADMIN', true);

/* That's all, stop editing! Happy blogging. */

You must set up Apache to use SSL of course.

A lot more info can be found here:

Administration Over SSL « WordPress Codex.

Get Internet Explorer 7 to work under Crossover Office in Ubuntu 12.10

2012-11-26T13:17:00+02:00

If IE7 just keeps opening the runonce3.aspx, you can use this to make it forget about it:

Disable ptrace security (please don't do this on a multi-user machine - see the codeweavers support wiki instructions regarding security):

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

To make it permanent, edit /etc/sysctl.d/10-ptrace.conf and set:

kernel.yama.ptrace_scope = 0

That will make other Windows programs run in Ubuntu as well. While the above is all that is needed to run IE7, you can disable the Run Once dialog as follows:

Make changes to the registry by opening the "Run a Windows Command" from the launcher, and running "regedit" in the bottle where you have IE7 installed.

This makes IE7 think the runonce page has been completed:

[HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main]
"RunOnceHasShown"=dword:00000001
"RunOnceComplete"=dword:00000001

While you're at it, this will change the default search page to Google:

[HKEY_CURRENT_USER\Software\Microsoft\Internet
Explorer\SearchScopes\{2FEDD0BC-4D55-413C-8B59-BFE70133A2CB}]
"DisplayName"="Google"
"URL"="http://www.google.com/search?q={searchTerms}"

Thanks go to:

Hello again, world!

2012-11-23T07:36:00+02:00

I'm just trying to get my old blog back online after neglecting it for a couple of years. And a disk crash. And no MySQL dumps, of course. Luckily I had the MyISAM data files so I was able to salvage most of it. Lot's of stuff not working yet. Working on it.

Cloning Ubuntu 10.04 Server KVM guests efficiently

2010-11-28T12:05:00+02:00

If you need to create lots of similar virtual machine guests running on QEMU/KVM, it is a very good idea to prepare a template guest image from which to clone the other guests. You should do whatever customizations you like before cloning. For instance I like to configure LVM and file systems to my liking, install openssh-server, install nfs-common and configure NFS mounts, install all available updates, add users or set up authentication, copy ssh keys, and do many other things so that they will be working out-of-the-box after cloning a number of guests from the template.

After you have installed and set-up your template virtual server to your liking, and would want to start cloning multiple instances of it, some tricks are needed to make things work more automatically after cloning and starting up the final copy.

If you simply clone a vanilla Ubuntu server installation multiple times, you will face some problems:

The network interfaces will be renamed at first boot because the MAC address will be different.
The ssh keys will be the same on each server (it works, but not the best idea)
The server hostname will be the same on each server
Statically configured IP address will be the same on each server, causing potential IP address conflicts

For IP addresses, I suggest using DHCP for the template image, so that each cloned guest will receive a unique IP during the first boot-up. You can then configure static IP addresses for the clones. I will show you how to do it almost automatically.

There are also a couple of optional tips which you should do for a virtual guest template before anything else.

This long post is divided into three parts:

Part I. Configuring the template guest image
Part II. Cloning and configuring multiple guests from your template
Part III. Conclusions and recommendations

Part I is the longest, but it will show you how to set up your template so that only one script must be run after guest boot-up to change network settings and hostname for each new guest. Also, I will show you how to make sure network interface naming is consistent from eth0 onwards, and that the host ssh keys will be unique on each host.

Part I. Configuring the template guest image

Let's start with the tips:

Tip 1: Make shutdown work

The shutdown command does not work by default from virt-manager nor virsh. To fix that, install acpid on the template guest image:

sudo apt-get install acpid

After that, shutdown should work from both virt-manager and virsh. For some strange reason, reboot works only from virt-manager, not virsh.

Tip 2: Make serial console work

This is very handy for console over ssh (no need for virt-manager, nor any kind of VNC connection for console). See earlier post:

{filename}/2010/serial-console-for-ubuntu-server-1004-kvm-guests.rst

Tip 3: Copy your ssh public key

It makes sense to copy your public ssh key at this point, so that you don't need to do it again for each guest. Simply go to your home directory on the template guest and run:

mkdir -p .ssh
chmod 700 .ssh
cd .ssh
cat >authorized_keys

Paste your public key there, and press Ctrl-D. Pasting does not work in a VNC console, but does work using the serial console trick mentioned in tip 2, or ssh connection.

Next, let's take a look at the problems mentioned in the start of the post:

Problem 1: Network interface naming

When Ubuntu boots up and detects a network interface with a MAC address which it has not seen before, it will take the next available eth number and add that to the udev rules file. The MAC addresses of network interfaces for cloned guests will be different from the template guest, so if you don't delete the corresponding lines from the file before cloning, the cloned guest's interface numbering will not start from eth0, but eth1 or something else depending on the number of interfaces you have configured.

In order to get your cloned guests' network interfaces start being numbered from eth0 onwards, simply delete the udev file which contains the rules for interface naming:

rm -f /etc/udev/rules.d/70-persistent-net.rules

That file will be regenerated at the first boot, so all clones will start their interface naming from eth0. But you must also remember that the next time you start up your template to make changes to it, it too will re-create the file. So remember to delete it every time you start up your template guest to make changes to it.

Problem 2: Individual SSH host keys

The ssh host keys are a bit more difficult thing. If you simply delete the keys from /etc/ssh/, they will not be regenerated automatically on Ubuntu. To make that happen, you need to run manually:

sudo dpkg-reconfigure openssh-server

In order to make that happen automatically during the first boot of a cloned guest, one line has to be added to the /etc/init/ssh.conf file (marked with bold):

/etc/init/ssh.conf:

respawn
respawn limit 10 5
umask 022
# replaces SSHD_OOM_ADJUST in /etc/default/ssh
oom never

pre-start script
    test -x /usr/sbin/sshd || { stop; exit 0; }
    test -e /etc/ssh/sshd_not_to_be_run && { stop; exit 0; }
    test -c /dev/null || { stop; exit 0; }

    # Regenerate ssh host keys if they are missing:
    test -f /etc/ssh/ssh_host_dsa_key || dpkg-reconfigure openssh-server

    mkdir -p -m0755 /var/run/sshd
end script

# if you used to set SSHD_OPTS in /etc/default/ssh, you can change the
# 'exec' line here instead
exec /usr/sbin/sshd

That will test the existence of host keys, and regenerate them if they are not present.

Before stopping your template guest, delete the ssh host keys:

rm -f /etc/ssh/*host*key*

Script to prepare template for cloning

I added a simple script, /usr/local/sbin/cloneprep, to ease the preparation of the template. It will simply delete your host ssh keys and udev persistent network rules (asking your permission first), and shut down the template guest afterwards so you can start cloning.

/usr/local/sbin/cloneprep:

#!/bin/sh

echo "TEMPLATE PREPARATION BEFORE CLONING"
echo
echo "This script will prepare this host for cloning. The network interface"
echo "name rules and ssh host keys will be deleted, but they will be"
echo "regenerated at the next boot (run sudo dpkg-reconfigure openssh-server"
echo "to regenerate ssh keys if needed)."
echo
echo "Are you sure you want to remove the ssh keys and network interface"
echo -n "rules on this host [y/N]? "

read YN

if [ "$YN" != "y" ]; then
  echo abort.
  exit 1
fi

echo
echo Removing persistent network interface rules...
rm -f /etc/udev/rules.d/70-persistent-net.rules

echo Removing host ssh keys...
rm -f /etc/ssh/*host*key*

echo
echo Your image is now ready for cloning. The machine will shut down.
echo

for i in 12 11 10 9 8 7 6 5 4 3 2 1; do echo "Halting machine in $i seconds (Ctrl-C to abort)"; sleep 1; done

halt

Give execute permission with "sudo chmod u+x /usr/local/sbin/cloneprep".

Remember to run cloneprep EVERY TIME you start up a template guest machine to change something. Don't use halt or shutdown, but run cloneprep instead.

It is not yet time to run it though, as we will do some further preparations to the template in the following section.

Problems 3 and 4: Hostname and IP address

To address these problems efficiently, we must create three of configuration files and a configuration script that will be run on each guest after the first boot.

To create a dozen guests, I created the following files under the template guest's /usr/local/etc.

/usr/local/etc/cloneaddresses:

This file will be used to look up the IP address for a given name.

guest01 192.168.1.101
guest02 192.168.1.102
guest03 192.168.1.103
guest04 192.168.1.104
guest05 192.168.1.105
guest06 192.168.1.106
guest07 192.168.1.107
guest08 192.168.1.108
guest09 192.168.1.109
guest10 192.168.1.110
guest11 192.168.1.111
guest12 192.168.1.112

(Use spaces instead of tabs in the cloneaddresses file. You can add as many spaces in between the names and addresses as you like, though.)

/usr/local/etc/clonehosts:

This file will be copied to /etc/hosts, with the GUESTNAME replaced with the guest's actual name. You can copy your actual /etc/hosts file to clonehosts and edit that.

127.0.0.1   localhost
127.0.1.1   GUESTNAME

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

/usr/local/etc/cloneinterfaces:

This file will be copied to /etc/network/interfaces, with the GUESTIP replaced with the guest's actual IP address. You can copy your actual /etc/network/interfaces file to cloneinterfaces and edit that.

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
  address GUESTIP
  netmask 255.255.255.0
  gateway 192.168.1.1

I created the configuration script as /usr/local/sbin/cloneconf, and it looks like this:

/usr/local/sbin/cloneconf:

#!/bin/sh

echo "GUEST CONFIGURATION AFTER CLONING"
echo

GUESTNAME=""

if [ "$1" = "" ]; then
  echo -n "What is the hostname of this guest? "
  read GUESTNAME
else
  GUESTNAME="$1"
fi

GUESTIP="$(cat /usr/local/etc/cloneaddresses | egrep ^$GUESTNAME  | awk '{print $2}')"

if [ "$GUESTIP" = "" ]; then
  echo "Hostname not found. Available hostnames:"
  echo
  cat /usr/local/etc/cloneaddresses
  exit 1
fi

echo "This script will configure the host with the following settings:"
echo Hostname: $GUESTNAME
echo IP address: $GUESTIP
echo
echo /etc/hostname:
echo $GUESTNAME
echo
echo /etc/hosts:
cat /usr/local/etc/clonehosts | sed -e "s/GUESTNAME/$GUESTNAME/g"
echo
echo /etc/network/interfaces:
cat /usr/local/etc/cloneinterfaces | sed -e "s/GUESTIP/$GUESTIP/g"
echo
echo -n "Are you sure you want to continue [y/N]? "

read YN
if [ "$YN" != "y" ]; then
  echo abort.
  exit 1
fi

echo $GUESTNAME > /etc/hostname
cat /usr/local/etc/clonehosts | sed -e "s/GUESTNAME/$GUESTNAME/g" > /etc/hosts
cat /usr/local/etc/cloneinterfaces | sed -e "s/GUESTIP/$GUESTIP/g" > /etc/network/interfaces
test -f /etc/ssh/ssh_host_dsa_key || dpkg-reconfigure openssh-server

echo
echo Done. The machine will now be rebooted to make changes effective.
echo
for i in 12 11 10 9 8 7 6 5 4 3 2 1; do echo "Rebooting in $i seconds (Ctrl-C to abort)"; sleep 1; done
reboot

Give execute permission with "sudo chmod u+x /usr/local/sbin/cloneconf".

That script must be run individually on each cloned guest machine. Either give the hostname as the single argument to the script, or the script will ask you for the hostname if you don't do that. The hostname must be found in the cloneaddresses file. The rest of the settings will be determined from the configuration files.

Part II. Cloning and configuring multiple guests from your template

Cloning guests from a template is as easy as:

Run the "cloneprep" script on the template guest and shut it down
Clone as many new guests as you wish with the virt-clone command
Start up each new guest
Run "cloneconf" on each new guest

Clone

One clone:

sudo virt-clone --original=template1 --name=guest01 --file=/var/lib/libvirt/images/guest01.img

Or a dozen clones:

for i in {01..12}; do sudo virt-clone -o template1 -n guest$i -f /var/lib/libvirt/images/guest$i.img; done

The virt-clone command will create a new xml configuration file under /etc/libvirt/qemu with a new UUID and randomly selected MAC address. Also, it will copy the image file to the location specified in the command line.

If you configured your template for serial console operation (link to the instructions in Tip 1 above), you can start up the new guest and get the console instantly using this command:

sudo virsh create /etc/libvirt/qemu/guest01.xml --console

To start your dozen guests simultaneously:

for i in {01..12}; do sudo virsh create /etc/libvirt/qemu/guest$i.xml; done

Then connect to the console of each individual guest with:

sudo virsh console guest01

You can get a list of running guests with "virsh list".

To end a serial console session, press Ctrl-]

Configure

The cloneconf script created in the first part of this post makes things easy.

After you have logged into a running guest machine (either VNC, serial console or ssh), run:

sudo cloneconf <hostname>

The hostname must be listed in the /usr/local/etc/cloneaddresses file created earlier. Otherwise the script will abort. It will show you changes it is about to make, and let you choose whether to commit them or not.

Reboot the guest after the script has run. Repeat for all clones.

The script will make changes to these files:

/etc/hostname
/etc/hosts
/etc/network/interfaces

It will replace the hostname and IP address with a suitable value.

Part III. Conclusions and recommendations

Customizing each cloned guest really just consists of changing or regenerating the following files:

/etc/hostname
/etc/hosts
/etc/network/interfaces
/etc/ssh/ssh_host_dsa_key
/etc/ssh/ssh_host_dsa_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pub
/etc/udev/rules.d/70-persistent-net.rules

After that, the rest of the template customization is up to you.

I recommend creating one master template with the instructions above, with very little installed software - the bare minimum you will install on ALL of your virtual machine guests. Configure all the authentication and NFS stuff, and whatever your environment requires.

Then clone this master template to create new templates for your web servers, mail servers etc. You can run cloneprep on them again even if you already ran cloneconf, to reverse a cloned guest back to a template (although change /etc/network/interfaces back to DHCP in order to avoid IP address conflicts).

After all this messing around with templates, deploying a new server is really quick and easy. You will have the right software installed, and nearly configured. And you can even customize the cloneconf script to configure whatever software you like.

I hope this helps!

Here you can download the scripts and configuration files in one package: clonetools.tar.gz (download no longer available)

Serial console for Ubuntu server 10.04 KVM guests

2010-11-27T13:15:00+02:00

The virt-manager VNC screen is fine for LAN connections, and good for running graphical sessions. X is not installed on Ubuntu server by default, and VNC is really bad over slow links even for text console. I like to configure serial console for all my virtualized guests, because with it, I can simply ssh into the virtual machine host, and run "virsh console <guest-name>" to get a working console. Very nice for fixing broken network connections or file systems, or any kind of boot problems. And I can do it using just my cell phone, ssh over 3G connection from anywhere!

In Ubuntu, there are three things you may want to configure to use serial console:

GRUB boot menu
Kernel and init messages
Pseudo tty to log in and get shell

Serial device for your virtual guest

But first things first: you must add a serial device for your guest machine in order to be able to use the serial console.

In virt-manager it is as easy as adding a new hardware device, type Serial, and device type "Pseudo TTY (pty)" into the machine configuration. No further configuration needed, just add the device and that's it.

In the libvirt XML configuration file the device should look something like this:

<serial type='pty'>
  <target port='0'/>
</serial>
<console type='pty'>
  <target port='0'/>
</console>

The GRUB boot menu, Linux kernel, and system init messages

With GRUB 1, it was possible to get the GRUB menu simultaneously over serial and graphical VNC console. Unfortunately I haven't found a way to do that with GRUB 2. Linux kernel fortunately does output messages both to the serial and VNC console if so configured. The upstart init will output its' messages to whatever the kernel is configured to do.

To make the GRUB menu and kernel messages show up over serial console, you need to edit the /etc/default/grub file and run update-grub afterwards. The changed or added lines are marked with bold below:

/etc/default/grub:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=
GRUB_HIDDEN_TIMEOUT_QUIET=false
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="text console=tty0 console=ttyS0,115200n8"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_LINUX_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Run update-grub after making the changes. During the next reboot, you will be able to choose your kernel using the GRUB menu, or run any grub commands, using the serial console.

The GRUB menu will show up only on the serial console, while the kernel and init messages will show up on both serial console and the graphical screen.

Detailed explanation

Here's a description of what each of the changes mean:

GRUB_HIDDEN_TIMEOUT=

This simply enables the GRUB menu, which is hidden by default in Ubuntu.

GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="text console=tty0 console=ttyS0,115200n8?

These two lines control the Linux kernel behaviour. This will append the necessary kernel command line options to all kernels, both normal and rescue mode. With the two console= options, the messages will go to both tty0 (VNC screen) and ttyS0 (serial console).

Also upstart init will show messages on both, and single user mode (runlevel 1) will work on both (the rescue mode root shell).

GRUB_TERMINAL=serial

GRUB_SERIAL_COMMAND=

N900 Scandinavic Letters from US keyboard

2009-12-20T14:46:00+02:00

My Nokia N900 has the US keyboard with four arrow keys but no diacritics, two of which are used in my native Finnish language. I actually like having the four arrow keys instead of two arrow keys plus dedicated diacritics. You can anyway get those from the on-screen keyboard, but it is better to remap some of the hardware keys to be able to punch them in quicker. Here's how.

To change the default hardware keyboard mapping, edit file /usr/share/X11/xkb/symbols/nokia_vndr/rx-51. This file describes the mapping from hardware keys to symbols on screen. You can change them how you like, but most of the keys are already mapped with symbols even with the modifier keys (Ctrl, Alt/Fn, Shift), and the corresponding symbol is printed in the hw button. So changing those might not be a good idea. However, the arrow keys on the US keyboard are free to use with the Alt/Fn modifier (the blue northeast pointing arrow on the left side of the keyboard). The arrow keys with the shift modifier key are used to select text, though, so they are not to be messed up.

The last stanza in the file is the arrows_4btns, which can be edited to provide more symbols directly from the keyboard. My configuration adds four important (to me) symbols: the letters '

Samba on AIX 5.3

2009-11-12T18:53:00+02:00

Here are instructions on how to get the pware Samba running on AIX 5.3.

1. Install these packages from the AIX installation CD:
- ldap.client.rte
- ldap.client.adt

Download these packages:

server:root>mkdir pware-samba
server:root>cd pware-samba
server:root>xargs wget -nd <<EOF http://pware.hvcc.edu/download/aix53-64/pware53-64.samba.3.4.2.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.base.5.3.0.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.cyrus-sasl.2.1.22.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.gettext.0.17.0.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.krb5.1.6.3.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.libiconv.1.13.1.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.ncurses.5.7.0.1.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.openldap.2.4.19.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.openssl.0.9.8.11.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.popt.1.10.4.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.bdb.4.7.25.4.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.rsync.3.0.6.0.bff.gz http://pware.hvcc.edu/download/aix53-64/pware53-64.zlib.1.2.3.0.bff.gz EOF
server:root>for i in *.gz; do gunzip $i; done

From: http://pware.hvcc.edu/download/aix53-64/

Install the packages.

The installed files go under /opt/pware64

Samba configuration file is /opt/pware64/lib/smb.conf

Extremely simple Samba configuration file:

[global]
  workgroup = MYGROUP
  log file = /var/log/%m.log
  max log size = 500

[myshare]
  comment = Sample share
  path = /tmp/myshare
  public = yes
  writable = yes
  browseable = yes

Enable POSIX Asynchronous IO if needed:

server:root>mkdev -l posix_aio0
posix_aio0 Available
server:root>lsdev -Cc posix_aio
posix_aio0 Available  Posix Asynchronous I/O

The posix_aio device is needed, otherwise all binaries will complain somewhat like this:

exec(): 0509-036 Cannot load program ./smbclient because of the following errors:
        0509-130 Symbol resolution failed for /usr/lib/libc.a[posix_aio_64.o] because:
        0509-136   Symbol _posix_kaio_rdwr64 (number 2) is not exported from
                   dependent module /unix.
        0509-136   Symbol _posix_listio64 (number 3) is not exported from
                   dependent module /unix.
        0509-136   Symbol _posix_acancel64 (number 4) is not exported from
                   dependent module /unix.
        0509-136   Symbol _posix_iosuspend64 (number 5) is not exported from
                   dependent module /unix.
        0509-136   Symbol _posix_aio_nwait (number 6) is not exported from
                   dependent module /unix.
        0509-136   Symbol _posix_aio_nwait64 (number 7) is not exported from
                   dependent module /unix.
        0509-136   Symbol _posix_aio_nwait_timeout (number 8) is not exported from
                   dependent module /unix.
        0509-136   Symbol _posix_aio_nwait_timeout64 (number 9) is not exported from
                   dependent module /unix.
        0509-136   Symbol _posix_iofsync64 (number 10) is not exported from
                   dependent module /unix.
        0509-026 System error: Error 0
        0509-192 Examine .loader section symbols with the
                 'dump -Tv' command.

So if you run into this error message, create the POSIX AIO device.

Run samba

server:root>/opt/pware64/sbin/smbd
server:root>/opt/pware64/sbin/nmbd

Add to inittab for automatic start at boot time

server:root>mkitab nmbd:2:once:/opt/pware64/sbin/nmbd
server:root>mkitab smbd:2:once:/opt/pware64/sbin/smbd

Links

http://pware.hvcc.edu/
http://pware.hvcc.edu/AIX-Samba.pdf
http://pware.hvcc.edu/download/
http://www.ibm.com/developerworks/aix/library/au-aix_samba/index.html

NIC bonding with Red Hat/CentOS

2009-10-21T17:17:00+03:00

Here are simple instructions on how to configure network interface bonding on Red Hat based distros. The thing I always forget. There's also a little script which will create a bonding interface bond0 between eth0 and eth1 and migrate existing IP settings from eth0. You can find it in the bottom of this post.

But first here's how to do it by hand.

Configure bonding module and interface alias:

/etc/modprobe.conf:

alias bond0 bonding
options bond0 miimon=100 mode=1

Insert bonding module:

# modprobe bonding

Configure network interfaces, two physical and one bonding interface:

/etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BOOTPROTO=static
HWADDR=00:11:22:33:44:55
ONBOOT=yes
TYPE=Ethernet
SLAVE=yes
MASTER=bond0

/etc/sysconfig/network-scripts/ifcfg-eth1:

DEVICE=eth1
BOOTPROTO=static
HWADDR=00:11:22:33:44:56
ONBOOT=yes
TYPE=Ethernet
SLAVE=yes
MASTER=bond0

/etc/sysconfig-network-scripts/ifcfg-bond0:

DEVICE=bond0
BOOTPROTO=none
IPADDR=192.168.0.2
NETMASK=255.255.255.0
GATEWAY=192.168.0.1
ONBOOT=yes

Restart networking (your network connections will drop at this point):

# /etc/init.d/network restart

That's it!

Automatic Script

Here's the script I promised. It takes your existing IP settings from eth0 and migrates them to a newly created bond0 which is created between eth0 and eth1:

cat >>/etc/modprobe.conf <<EOF
alias bond0 bonding
options bond0 miimon=100 mode=1
EOF

cat >>/etc/sysconfig/network-scripts/ifcfg-bond0 <<EOF
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
EOF

cat /etc/sysconfig/network-scripts/ifcfg-eth0 | \
  egrep 'IPADDR|NETMASK|GATEWAY' >>/etc/sysconfig/network-scripts/ifcfg-bond0

HWADDR_ETH0="$(cat /etc/sysconfig/network-scripts/ifcfg-eth0 | grep HWADDR)"
HWADDR_ETH1="$(cat /etc/sysconfig/network-scripts/ifcfg-eth1 | grep HWADDR)"

cat > /etc/sysconfig/network-scripts/ifcfg-eth0 <<EOF
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
SLAVE=yes
MASTER=bond0
$HWADDR_ETH0
EOF

cat > /etc/sysconfig/network-scripts/ifcfg-eth1 <<EOF
DEVICE=eth1
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
SLAVE=yes
MASTER=bond0
$HWADDR_ETH1
EOF

More information can be found in the bonding.txt of the kernel source documentation.

http://www.netmite.com/android/mydroid/kernel/Documentation/networking/bonding.txt

Redundant iSCSI storage for Linux

2009-06-10T18:14:00+03:00

Here's how to set up relatively cheap redundant iSCSI storage on Linux. The redundancy is achieved using LVM mirroring, and the storage servers consist of commodity hardware, running the OpenFiler Linux distribution, which expose their disks to the clients using iSCSI over Ethernet. The servers are completely separate entities, and the purpose of this mirroring is to keep the logical volumes available, even while one of the storage servers is down for maintenance or due to hardware failure.

Ultimately the disks of the iSCSI target servers will show up as normal SCSI disks on the client (/dev/sdb, /dev/sdc, ...). The data will be moved across the network transparently. It is preferable to use multiple gigabit network interface cards on both the initiator and the target, and bond them together for reliability and speed gain (or use Device Mapper Multipath). A separate VLAN for iSCSI traffic is recommended for security and speed. By default, the traffic is not encrypted so your disk blocks can easily be sniffed using tcpdump.

I created identical logical volumes on both OpenFiler servers and mapped them to iSCSI targets. The iSCSI initiator (client) here is an Ubuntu 9.04 desktop.

Install Open-iSCSI and map targets

On the client, install Open-iSCSI.

aptitude install open-iscsi

Run the discovery to see available targets (the IP address is the address of one of the servers).

iscsiadm -m discovery -t st -p 192.168.1.115

You should get a target list as the output.

192.168.1.115:3260,1 iqn.2006-01.com.openfiler:linuxtest1lv

Map the target to a SCSI disk.

iscsiadm -m node -T iqn.2006-01.com.openfiler:linuxtest1lv -p 192.168.1.115 --login

dmesg should now show a that a new SCSI disk was detected.

[600584.938727] scsi 2:0:0:0: Direct-Access     OPNFILER VIRTUAL-DISK     0    PQ: 0 ANSI: 4
[600584.947903] sd 2:0:0:0: [sdb] 4194304 512-byte hardware sectors: (2.14 GB/2.00 GiB)
[600584.983070] sd 2:0:0:0: [sdb] Write Protect is off
[600584.983074] sd 2:0:0:0: [sdb] Mode Sense: 77 00 00 08
[600584.988064] sd 2:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
[600584.989379] sd 2:0:0:0: [sdb] 4194304 512-byte hardware sectors: (2.14 GB/2.00 GiB)
[600584.989974] sd 2:0:0:0: [sdb] Write Protect is off
[600584.989977] sd 2:0:0:0: [sdb] Mode Sense: 77 00 00 08
[600584.991359] sd 2:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
[600584.991363]  sdb: unknown partition table
[600585.008012] sd 2:0:0:0: [sdb] Attached SCSI disk
[600585.008072] sd 2:0:0:0: Attached scsi generic sg2 type 0

You can now use the disk as a normal SCSI disk.

Discover the second storage server.

iscsiadm -m discovery -t st -p 192.168.1.120

Target found:

192.168.1.120:3260,1 iqn.2006-01.com.openfiler:linuxtest1lv-2

Map the target.

iscsiadm -m node -T iqn.2006-01.com.openfiler:linuxtest1lv-2 192.168.1.120 --login

Make persistent across reboots

The discovered nodes will automatically show up under /etc/iscsi/nodes. If you wish to make them available automatically after reboot, change the following line in the corresponding node file:

node.conn[0].startup = manual

Change to:

node.conn[0].startup = automatic

Partition with fdisk (optional)

I partitioned the disks with fdisk. This is optional, but I like to do it because it makes easier to detect the type of the disk just by checking the partition table.

Disk /dev/sdb: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Disk identifier: 0x32d429c4

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1009     2095662   8e  Linux LVM

Disk /dev/sdc: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders
Units = cylinders of 4154 * 512 = 2126848 bytes
Disk identifier: 0x9823ed68

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1        1009     2095662   8e  Linux LVM

The LVM Part

Install Logical Volume Manager.

aptitude install lvm2

Create physical volumes and the volume group.

pvcreate /dev/sdb1
pvcreate /dev/sdc1
vgcreate vg0 /dev/sdb1 /dev/sdc1

Create a mirrored logical volume.

lvcreate --mirrors 1 --corelog --name testlv --size 512M vg0

Create filesystem and mount.

mke2fs -j /dev/vg0/testlv
mount /dev/vg0/testlv /mnt/test

Speed

Test read speeds.

hdparm -t /dev/mapper/vg0-testlv

10 MB per second is about the max I can get with this test system which uses 100 Mbit/s ethernet.

/dev/mapper/vg0-testlv:
 Timing buffered disk reads:   32 MB in  3.22 seconds =   9.95 MB/sec

On a production system, gigabit is a must (preferably multiple links bonded).

Status of the Mirrored Logical Volume

To check the status of the mirrored logical volume, run the command "lvs":

LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
testlv vg0  mwi-ao 512,00M                        100,00

The Copy% will show the percentage of copied extents. 100% indicates the mirrors are synced. Whenever a mirror is out-of-sync and is being updated, the percentage will be less.

The commands "lvdisplay -m" and "pvdisplay -m" will show you a detailed map of the extents on the physical volumes:

lvdisplay -m

--- Logical volume ---
LV Name                /dev/vg0/testlv
VG Name                vg0
LV UUID                5ookbu-qJ9h-rzBA-D6Ek-mkH2-Vryc-EYYqvp
LV Write Access        read/write
LV Status              available
# open                 1
LV Size                512,00 MB
Current LE             128
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           252:2

--- Segments ---
Logical extent 0 to 127:
  Type        mirror
  Mirrors     2
  Mirror size     128
  Mirror region size  512,00 KB
  Mirror original:
    Logical volume    testlv_mimage_0
    Logical extents   0 to 127
  Mirror destinations:
    Logical volume    testlv_mimage_1
    Logical extents   0 to 127

pvdisplay -m

--- Physical volume ---
PV Name               /dev/sdb1
VG Name               vg0
PV Size               2,00 GB / not usable 2,54 MB
Allocatable           yes
PE Size (KByte)       4096
Total PE              511
Free PE               383
Allocated PE          128
PV UUID               JINpaF-WiCp-sEH2-2PcK-bEvR-ht8j-mRAg05

--- Physical Segments ---
Physical extent 0 to 127:
  Logical volume  /dev/vg0/testlv_mimage_0
  Logical extents 0 to 127
Physical extent 128 to 510:
  FREE

--- Physical volume ---
PV Name               /dev/sdc1
VG Name               vg0
PV Size               2,00 GB / not usable 2,54 MB
Allocatable           yes
PE Size (KByte)       4096
Total PE              511
Free PE               383
Allocated PE          128
PV UUID               V7dMTV-gWLe-gRWy-H7LU-7mwI-LsBu-2uIC7C

--- Physical Segments ---
Physical extent 0 to 127:
  Logical volume  /dev/vg0/testlv_mimage_1
  Logical extents 0 to 127
Physical extent 128 to 510:
  FREE

Testing for failure

When the other iSCSI server was brought down, it took about two minutes before the iSCSI initiator gave up. After this, the mounted volume was working without problems. During the two-minute timeout countdown, some slowness and waiting was experienced.

After the iSCSI server was brought up again, the other half of the mirror was restored and synced automatically. In conclusion, I would say my mirrored logical volume can be thought of as highly available.

It seems that the timeout value can be set in the node configuration file (although I didn't test it):

node.session.timeo.replacement_timeout = 120

Links and references

Differential Xcopy parameters

2009-04-15T17:17:00+03:00

I always forget what the correct options for the Windows xcopy command are when I simply want to synchronize one directory over another one, so that only changed files are overwritten. To do it well, you need to remember too many options. And robocopy is never installed when you need it.

Here it is:

xcopy /keychord source destination

The options:

/k copy attributes
/e be recursive, and even re-create empty directories
/y say yes to prompts
/c don't stop on errors
/h copy hidden and system files
/o copy ownership and ACL information
/r overwrite read-only files
/d copy only changed files (or new files)

Just remember, "Key Chord"

OpenSSH public key authentication

2009-01-09T18:11:00+02:00

First, create a key-pair with ssh-keygen. This is a one-time operation.

ssh-keygen -t dsa

It is good practice to enter a good password, but you may also leave the password empty. That will leave your private key vulnerable to local attacks, but if you need to login somewhere from a cron job, you probably need to do that.

For interactive logins, it is better to use ssh-agent.

Copy your public key to the server

Newer distributions have the ssh-copy-id command, which makes copying your key as easy as:

ssh-copy-id remoteserver

If you don't have ssh-copy-id, you must manually append your public key to the ".ssh/authorized_keys" file under your home directory on the remote host. This does just that:

cat ~/.ssh/id_dsa.pub | ssh remoteserver 'cat >>.ssh/authorized_keys'

Server side sshd configuration

This is what you need in "/etc/ssh/sshd_config" for public key authentication to work in the first place:

PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

A restart is required if you changed anything.

Allowing root to log in with RSA/DSA keys

To enable public key authentication as root, the "PermitRootLogin" configuration setting must not be set to "no". In other words, it should either be set to:

PermitRootLogin yes

or the way I prefer it:

PermitRootLogin without-password

The latter form will disallow root login with a password, but allow it with a key. Makes brute force root password guessing impossible.

Slow authentication

If login takes a long time (more than about 5 seconds), make sure ssh does not try to make a reverse lookup for the remote host IP address. You will not detect this unless you yank the "LogLevel" to "Debug" and restart ssh. After that, you will see messages like this while the client is waiting for login:

Jan  9 14:47:31 localhost sshd[26374]: debug3: Trying to reverse map address 192.168.0.1.

There are two things that may cause the reverse lookups. The first is a configuration file option in "/etc/ssh/sshd_config" called "VerifyReverseMapping". It should be set to "no":

VerifyReverseMapping no

That is also the default setting, so it is less likely to be the real reason for reverse lookups. The following one is the more likely one.

The second thing that may cause OpenSSH to do reverse lookups is that by default it will try to store the remote hostname in utmp. To disable this, it needs the option "-u0" on the command line.

On Red Hat(ish) systems you can do this easily by adding the following option to the "/etc/sysconfig/sshd" file:

OPTIONS="-u0"

On Ubuntu, add this to "/etc/default/ssh":

SSHD_OPTS="-u0"

Restart sshd, and you will have a faster login. This setting will also speed up logins when the reverse lookups do work, because they will in any case slow down the authentication process.

Ubuntu 8.10 on Thinkpad X300

2008-12-08T23:22:00+02:00

I upgraded my Lenovo Thinkpad X300 to Ubuntu 8.10 Intrepid Ibex today. My original installation notes for Hardy are here.

After the upgrade, sound was working without compiling an ALSA snapshot by hand. Also, 3G connections worked straight out of the NetworkManager applet, which is very nice. But WiFi was broken, as the nm-applet refused to connect to any SSID, encrypted or open. That seemed to be due to myself using the development networkmanager packages with Hardy. They were not upgraded correctly. The remedy was to remove all network-manager packages and reinstall them.

The xorg.conf mouse settings don't work on 8.10 anymore. Instead, HAL needs to be configured like this. Create file "/etc/hal/fdi/policy/mouse-wheel.fdi" with the following content:

<match key="info.product" string="TPPS/2 IBM TrackPoint">
 <merge key="input.x11_options.EmulateWheel" type="string">true</merge>
 <merge key="input.x11_options.EmulateWheelButton" type="string">2</merge>
 <merge key="input.x11_options.YAxisMapping" type="string">4 5</merge>
 <merge key="input.x11_options.XAxisMapping" type="string">6 7</merge>
 <merge key="input.x11_options.Emulate3Buttons" type="string">true</merge>
 <merge key="input.x11_options.EmulateWheelTimeout" type="string">200</merge>
</match>

Reboot the computer, and you should have middle button scrolling working as always. Here's the link: http://www.thinkwiki.org/wiki/How_to_configure_the_TrackPoint#TrackPoint_under_Ubuntu_8.10_using_HAL

If you want to upgrade to OpenOffice.org 3.0, here's a nice howto:

http://forums.ubuntu.com.my/viewtopic.php?f=42&t=616

VMWare Workstation 6.5 worked pretty much fine after the upgrade. The modules were compiled correctly. The only annoyance was that some keyboard keys seem to be mapped in a wrong way (perhaps something to do with the new X server version). Anyway, there's the fix: http://communities.vmware.com/thread/177133. All you need to do i add the following key mappings to /etc/vmware/config and restart:

xkeymap.keycode.108 = 0x138 # Alt_R
xkeymap.keycode.106 = 0x135 # KP_Divide
xkeymap.keycode.104 = 0x11c # KP_Enter
xkeymap.keycode.111 = 0x148 # Up
xkeymap.keycode.116 = 0x150 # Down
xkeymap.keycode.113 = 0x14b # Left
xkeymap.keycode.114 = 0x14d # Right
xkeymap.keycode.105 = 0x11d # Control_R
xkeymap.keycode.118 = 0x152 # Insert
xkeymap.keycode.119 = 0x153 # Delete
xkeymap.keycode.110 = 0x147 # Home
xkeymap.keycode.115 = 0x14f # End
xkeymap.keycode.112 = 0x149 # Prior
xkeymap.keycode.117 = 0x151 # Next
xkeymap.keycode.78 = 0x46 # Scroll_Lock
xkeymap.keycode.127 = 0x100 # Pause
xkeymap.keycode.133 = 0x15b # Meta_L
xkeymap.keycode.134 = 0x15c # Meta_R
xkeymap.keycode.135 = 0x15d # Menu

Installing PHP 5 on AIX using IBM HTTP Server

2008-12-08T16:48:00+02:00

I was not able to compile PHP 5.2.6 with IBM HTTP Server 6.1 as a module, so I compiled it as a CGI binary instead. Here's how to do it.

1. Download the AIX toolbox from:

ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/

You don't need to download everything, but that is the easiest way. The whole toolbox is about 2.8 GB in size. If you have wget, just say:

wget ftp://ftp.software.ibm.com/aix/freeSoftware/aixtoolbox/RPMS/ppc/

2. Install GCC and the rest of the GNU toolchain

Go to the toolbox dir and run the following command.

xargs rpm -iv << EOF
autoconf/autoconf-2.59-1.aix5.1.noarch.rpm
automake/automake-1.8.5-1.aix5.1.noarch.rpm
binutils/binutils-2.14-3.aix5.1.ppc.rpm
gcc/gcc-4.2.0-3.aix5.3.ppc.rpm
gcc/gcc-cplusplus-4.2.0-3.aix5.3.ppc.rpm
gcc/gcc-locale-4.2.0-3.aix5.3.ppc.rpm
gcc/libgcc-4.2.0-3.aix5.3.ppc.rpm
gcc/libstdcplusplus-4.2.0-3.aix5.3.ppc.rpm
gcc/libstdcplusplus-devel-4.2.0-3.aix5.3.ppc.rpm
gdbm/gdbm-1.8.3-2.aix5.1.ppc.rpm
gdbm/gdbm-devel-1.8.3-2.aix5.1.ppc.rpm
libtool/libtool-1.5.8-2.aix5.1.ppc.rpm
m4/m4-1.4.1-1.aix5.1.ppc.rpm
make/make-3.80-1.aix5.1.ppc.rpm
EOF

You will probably want to replace the package names with the newest versions.

3. Install PHP 5 prerequisites

In the same toolbox dir, run:

xargs rpm -iv << EOF
bzip2/bzip2-1.0.2-4.aix5.1.ppc.rpm
gd/gd-1.8.4-3.aix5.1.ppc.rpm
gd/gd-devel-1.8.4-3.aix5.1.ppc.rpm
gd/gd-progs-1.8.4-3.aix5.1.ppc.rpm
gettext/gettext-0.10.40-8.aix5.2.ppc.rpm
libpng/libpng-1.2.8-8.aix5.2.ppc.rpm
libpng/libpng-devel-1.2.8-8.aix5.2.ppc.rpm
libjpeg/libjpeg-6b-6.aix5.1.ppc.rpm
libjpeg/libjpeg-devel-6b-6.aix5.1.ppc.rpm
freetype/freetype-1.3.1-9.aix5.1.ppc.rpm
freetype/freetype-devel-1.3.1-9.aix5.1.ppc.rpm
freetype2/freetype2-2.1.7-5.aix5.1.ppc.rpm
freetype2/freetype2-devel-2.1.7-5.aix5.1.ppc.rpm
libxml2/libxml2-2.6.21-3.aix5.2.ppc.rpm
libxml2/libxml2-devel-2.6.21-3.aix5.2.ppc.rpm
zlib/zlib-1.2.3-4.aix5.2.ppc.rpm
zlib/zlib-devel-1.2.3-4.aix5.2.ppc.rpm
EOF

This really depends on what you want to compile in. My settings will require the stuff listed above.

Also, install fileset bos.adt.libm from the AIX installation media.

4. Download and unpack the latest PHP 5

Download the latest version from:

http://www.php.net/downloads.php#v5

I downloaded 5.2.6. Unpack:

gunzip php-5.2.6.tar.gz
tar xvf php-5.2.6.tar
cd php-5.2.6

5. Configure PHP 5

The AIX toolbox packages have been compiled with a default prefix of /opt/freeware, and that is where all the files from the packages are installed. As we compile php from scratch, it is good practice to configure the php package to go under /usr/local (see the "prefix" below). The "with-config-file-path" sets the the place where PHP searches for its' configuration file.

export PATH=/opt/freeware/bin:$PATH

./configure \
  --prefix=/usr/local \
  --with-config-file-path=/usr/IBMIHS/conf/php.ini \
  --enable-shared \
  --disable-static \
  --enable-maintainer-zts \
  --enable-calendar \
  --enable-bcmath \
  --enable-sockets \
  --enable-zip \
  --with-gd \
  --with-zlib \
  --with-libxml-dir=/opt/freeware \
  --with-zlib-dir=/opt/freeware \
  --with-bz2 \
  --with-gettext=/opt/freeware \
  --with-jpeg-dir=/opt/freeware \
  --with-png-dir=/opt/freeware \
  --with-freetype-dir=/opt/freeware

6. Compile

Compile:

make

You may run into the following errors while compiling:

cc1: out of memory allocating XXX bytes after a total YYY of bytes

This is a ulimit issue. Rise the soft limits to overcome it:

ulimit -S -s 3276800    # stack
ulimit -S -m 13107200  # memory
ulimit -S -d 13107200  # data area

execvp: /bin/sh: The parameter or environment lists are too long.

The parameter list to a command is too long. This limit can be raised. Check out the size with this command:

lsattr -El sys0 -a ncargs

Rise it with this command:

chdev -l sys0 -a ncargs=60

7. Test

make test

8. Install

make install
cp php.ini-recommended /usr/IBMIHS/conf/php.ini

9. Configure IHS

I added the following lines to httpd.conf:

ScriptAlias /php5-cgi /usr/local/bin/php-cgi
Action php-cgi /php5-cgi
AddHandler php-cgi .php

Also, to make index.php work, modify the DirectoryIndex directive:

DirectoryIndex index.html index.html.var index.php

10. Test that everything works

Test the configuration with:

/usr/IBMIHS/bin/apachectl -t

And restart IHS to make the new configuration effective:

/usr/IBMIHS/bin/apachectl restart

Links

Using the GNU C/C++ compiler on AIX
http://www.ibm.com/developerworks/aix/library/au-gnu.html
- The authors explain why you should use GCC compiler, which compiler options are specific to pSeries, what you need to know about shared libraries, and common gotchas and solutions.

IBM AIX Toolbox download information

http://www-03.ibm.com/systems/power/software/aix/linux/toolbox/download.html

Hosting PHP Applications on the IBM HTTP Server

http://www.ibm.com/developerworks/opensource/library/os-phphttp/

Raising the command line and environment limit:

http://www.ibm.com/developerworks/forums/thread.jspa?threadID=170563

SSH tunneling your way through multiple gateways

2008-10-19T17:56:00+03:00

Ths SSH protocol supports tunneling arbitrary ports from your local host to a remote network that is only reachable through a remote gateway machine. The typical situation is that you have a, say, web server in a network which is only accessible from inside the network. If you have an ssh gateway machine within the network, you can get to the web server using tunneling.

A Simple Tunnel

This is what a typical one-gateway tunneling looks like:

Here's how to set it up:

mkortela@laptop:~ $ ssh -L 8080:webserver:80 gw
mkortela@gw:~ $

Now I can connect to my laptop's localhost:8080 and access the webserver's content. I chose port 8080 because port 80 is a restricted port (you need to be root to listen to ports under 1024). Also, make sure to choose a port which is not in use.

Multiple Gateways

If there are multiple gateway hosts between local host and the webserver, the solution becomes a little bit more tricky. I have to make sure that the tunnel goes all the way from my laptop to the last gateway, which then forwards the connection to the target web server.

mkortela@laptop:~ $ ssh -L 1234:localhost:1234 gw1
mkortela@gw1:~ $ ssh -L 1234:webserver:80 gw2
mkortela@gw2:~ $

Now I am be able to connect to my laptop's localhost:1234, which is tunneled over two ssh tunnels to webserver:80.

A shorter form:

mkortela@laptop:~ $ ssh -L 1234:localhost:1234 gw1 "ssh -L 1234:server:1234 gw2"

You can add as many hops as you like.

mkortela@laptop:~ $ ssh -L 1234:localhost:1234 gw1
mkortela@gw1:~ $ ssh -L 1234:localhost:1234 gw2
mkortela@gw2:~ $ ssh -L 1234:localhost:1234 gw3
mkortela@gw3:~ $ ssh -L 1234:webserver:80 gw4
mkortela@gw4:~ $

Just make sure the port you choose is not in use on any machine. You can use a different port for each hop if you like.

Dynamic IPv6 routing with Cisco IOS and Quagga on OpenWRT

2008-10-19T17:46:00+03:00

Here's how to make dynamic IPv6 routing work between a Cisco IOS router and an OpenWRT Linux Quagga router. I couldn't find a similar howto anywhere, so I decided to write my own.

I am using OpenWRT Kamikaze 7.09 (kernel 2.4) on an ASUS WL-500gP wireless router. Any IPv6 enabled Cisco router should do.

I assume you have already installed the IPV6 kernel modules and userland tools, and set up static addresses for your interfaces (if you haven't check out the OpenWRT IPv6 Howto).

I am using SixXS for tunneling an IPv6 /48 prefix over IPv4. Here's a diagram of my setup:

Install Quagga

To install Quagga, add the Kamikaze 7.06 repository (the 7.09 repository does not have Quagga yet).

root@OpenWrt:~# echo "src oldrelease http://downloads.openwrt.org/kamikaze/7.06/brcm47xx-2.6/packages" >>/etc/ipkg.conf
root@OpenWrt:~# ipkg update
root@OpenWrt:~# ipkg install quagga
root@OpenWrt:~# ipkg install quagga-libzebra
root@OpenWrt:~# ipkg install quagga-ripngd

If you want to use OSPFv3 for IPv6, install also the ospf6d package:

root@OpenWrt:~# ipkg install quagga-ospf6d

In this howto, we will use the ripngd daemon because it is easier to set up and good enough for our small network.

Configure Zebra

To create an initial configuration file for the Zebra daemon, create a /etc/quagga/zebra.conf:

!
! Zebra configuration file
!
hostname wl1
password zebra
enable password zebra
!
log stdout
!
!

You should, of course, customize your passwords (and hostname). Then start Quagga daemons:

root@OpenWrt:~# /etc/init.d/quagga start
quagga.init: Starting zebra ... done.
quagga.init: Starting watchquagga ... done.

Two new processes should be running, the Zebra daemon and the "watchguagga" daemon:

5083 quagga      852 S   /usr/sbin/zebra -d
5088 root        424 S   /usr/sbin/watchquagga -d -z -T 60 -R /usr/sbin/quagga.init watchrestart zebra

You can now telnet to the default port:

root@OpenWrt:~# telnet localhost 2601

Entering character mode
Escape character is '^]'.

Hello, this is Quagga (version 0.98.6).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

User Access Verification

Password:
wl1> ?
  echo      Echo a message back to the vty
  enable    Turn on privileged mode command
  exit      Exit current mode and down to previous mode
  help      Description of the interactive help system
  list      Print command list
  quit      Exit current mode and down to previous mode
  show      Show running system information
  terminal  Set terminal line parameters
  who       Display who is on vty
wl1>

Very Ciscoish, eh? Although the command set is far from "complete", many commands work like they do on Cisco IOS:

wl1> show ip forwarding
IP forwarding is on
wl1> show ipv6 forwarding
ipv6 forwarding is off
wl1> enable
Password:
wl1# conf t
wl1(config)# ipv6 forwarding
wl1# write mem
Configuration saved to /etc/quagga//zebra.conf

Configure ripngd

The Zebra daemon does not handle any dynamic routing, it just works as a routing information redistributor. It will forward routing information between different dynamic routing daemons (ospf, rip, bgp) and the kernel routing table. Whenever the ripng daemon detects a change in the network's routing configuration, it will hand down the change to Zebra, which in turn will make a modification to the kernel routing table (which is the actual, effective routing table). If you have another routing daemon process running, Zebra will also notify that daemon about the change.

To configure ripngd, create an initial config:

root@OpenWrt:~# echo hostname wl1 > /etc/quagga/ripngd.conf
root@OpenWrt:~# echo router ripng >> /etc/quagga/ripngd.conf
root@OpenWrt:~# echo password zebra >> /etc/quagga/ripngd.conf
root@OpenWrt:~# echo enable password zebra >> /etc/quagga/ripngd.conf

Then restart quagga:

root@OpenWrt:~# /etc/init.d/quagga restart
quagga.init: Stopping watchquagga ... killed 11576 ... done.
quagga.init: Stopping zebra ... killed 11571 ... done.
quagga.init: Starting zebra ... done.
quagga.init: Starting ripngd ... done.
quagga.init: Starting watchquagga ... done.

Configure dynamic routing with RIP:

root@OpenWrt:~# telnet localhost 2603

Entering character mode
Escape character is '^]'.

Hello, this is Quagga (version 0.98.6).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

User Access Verification

Password:
wl1> enable
Password:
wl1# conf t
wl1(config)# router ripng
wl1(config-router)# network br-lan
wl1(config-router)# network eth0.1
wl1(config-router)# end
wl1# wr mem
Configuration saved to /etc/quagga//ripngd.conf

Cisco IOS Configuration

Next we will have to configure the other participant of the dynamic routing process. I am using a Cisco 877W with Advanced IP Services (you don't get IPv6 support below that level).

Cisco IOS Configuration:

cisco#conf t
cisco(config)#ipv6 router rip ripng
cisco(config-rtr)#redistribute connected
cisco(config-rtr)#redistribute static
cisco(config)#interface vlan 1
cisco(config-if)#ipv6 rip ripng enable

That's it! After a while you should see the routes propagated between the two routers:

root@OpenWrt:~# ip -f inet6 route
2001:db8:100:100::/64 via fe80::21c:eff:fed9:32ef dev eth0.1  proto zebra  metric 2  mtu 1500 advmss 1440
2001:db8:0:1::/64 dev eth0.1  proto kernel  metric 256  expires 2584987sec mtu 1500 advmss 1440
2001:db8:0:3::/64 dev br-lan  metric 256  mtu 1500 advmss 1440
2000::/3 via fe80::21c:eff:fed9:32ef dev eth0.1  proto zebra  metric 2  mtu 1500 advmss 1440
fe80::/64 dev eth0  metric 256  mtu 1500 advmss 1440
fe80::/64 dev eth0.0  metric 256  mtu 1500 advmss 1440
fe80::/64 dev br-lan  metric 256  mtu 1500 advmss 1440
fe80::/64 dev eth0.1  metric 256  mtu 1500 advmss 1440
fe80::/64 dev wl0  metric 256  mtu 1500 advmss 1440
ff00::/8 dev eth0  metric 256  mtu 1500 advmss 1440
ff00::/8 dev eth0.0  metric 256  mtu 1500 advmss 1440
ff00::/8 dev br-lan  metric 256  mtu 1500 advmss 1440
ff00::/8 dev eth0.1  metric 256  mtu 1500 advmss 1440
ff00::/8 dev wl0  metric 256  mtu 1500 advmss 1440
unreachable default dev lo  proto none  metric -1  error -128

cisco#show ipv6 route
IPv6 Routing Table - 8 entries
Codes: C - Connected, L - Local, S - Static, R - RIP, B - BGP
       U - Per-user Static route
       I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea, IS - ISIS summary
       O - OSPF intra, OI - OSPF inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
       D - EIGRP, EX - EIGRP external
S   2000::/3 [1/0]
     via 2001:14B8:100:1DB::1
C   2001:db8:100:100::/64 [0/0]
     via ::, Tunnel0
L   2001:db8:100:100::2/128 [0/0]
     via ::, Tunnel0
C   2001:db8:0:1::/64 [0/0]
     via ::, Vlan1
L   2001:db8:0:1::1/128 [0/0]
     via ::, Vlan1
R   2001:db8:0:2::/64 [120/2]
     via FE80::217:31FF:FED6:983E, Vlan1
L   FE80::/10 [0/0]
     via ::, Null0
L   FF00::/8 [0/0]
     via ::, Null0

Sendmail relay configuration on AIX

2008-09-18T11:32:00+03:00

This document describes how to set up a Sendmail e-mail gateway or relay which will be able to process incoming mail and route it to different mail servers based on domain information. The routing table is based on the Sendmail mailertable feature instead of the usual MX record based routing. This will come handy when there is a need to route mail internally in a different way than externally.

AIX configuration

Make sure "bos.net.tcp.adt" is installed.

The file /usr/samples/tcpip/sendmail/cf/aixsample.mc describes a minimal sample configuration. That can be used as the basis of your configuration. Please note however that there are some dangerous default settings in that file which must be changed before deployment.

Sendmail configuration: the sendmail.mc

I made my /etc/mail/sendmail.mc look like this:

divert(0)dnl
OSTYPE(aixsample)dnl
FEATURE(`mailertable',`dbm /etc/mail/mailertable')dnl
FEATURE(virtusertable)dnl
FEATURE(domaintable)dnl
FEATURE(no_default_msa)
FEATURE(relay_hosts_only)dnl
FEATURE(access_db)
FEATURE(`greet_pause',5000)dnl
DOMAIN(generic)dnl
define(`confSMTP_LOGIN_MSG', `$j Sendmail $b')
define(`confPRIVACY_FLAGS',`authwarnings,novrfy,noexpn,noverb')dnl
define(`confBAD_RCPT_THROTTLE', `1')dnl
define(`confCONNECTION_RATE_THROTTLE', `100')dnl
MAILER(local)dnl
MAILER(smtp)dnl
MAILER(uucp)
DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl

To process the .mc file into a proper Sendmail configuration file, sendmail.cf, use the following commands:

cd /usr/samples/tcpip/sendmail/m4/
m4 cf.m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

Security

I removed three lines and added one to the sample configuration file for security before deploying it.

I removed the lines:

FEATURE(accept_unresolvable_domains)dnl
FEATURE(accept_unqualified_senders)dnl
FEATURE(promiscuous_relay)dnl

promiscuous_relay

This is important! Leaving the promiscuous_relay feature on will make your server an open relay and thus a good host for spammers to do what they do. It *being a default setting is quite unbelievable. **So remember to remove it.*

There is usually no need to keep the other two options, accepting unresolvable domains or unqualified senders. Those will just close some holes form spammers.

The lines I added to the sample configuration are:

FEATURE(relay_hosts_only)dnl
FEATURE(access_db)
FEATURE(`greet_pause',5000)dnl
define(`confPRIVACY_FLAGS',`authwarnings,novrfy,noexpn,noverb')dnl
define(`confBAD_RCPT_THROTTLE', `1')dnl
define(`confCONNECTION_RATE_THROTTLE', `100')dnl

greet_pause

The greet_pause feature makes sendmail wait for a specified amount of time at the beginning of an incoming connection, before greeting the connecting party. A value of 5000 means 5 seconds. Proper mail clients will wait for the greeting, while many spambots start flooding immediately. They will be discarded by this rule.

authwarnings

The authwarnings privacy flag tells sendmail to insert X-Authentication-Warnings: headers into the mail whenever it suspects that the message is not authentic.

novrfy

The novrfy privacy flag will deny email address verification queries (the VRFY allows anyone to check if a user exists in your mail server - this will prevent it).

noexpn

The noexpn privacy flag will deny mailing list queries (the EXPN allows anyone to check who belongs to your mailing lists - this will precent it).

noverb

The noverb privacy flag will deny requests for verbose mode, which might reveal information about your installation to an outsider.

BAD_RCPT_THROTTLE

The BAD_RCPT_THROTTLE setting will add an additional annoyance for spammers. If a specified number of recipients in a single SMTP transaction have been rejected, sendmail will sleep for one second after each subsequent RCPT command in that transaction. That will make it impractical to try and guess usernames. Legitimate mail will get delivered, though.

CONNECTION_RATE_THROTTLE

The CONNECTION_RATE_THROTTLE Sets the maximum number of connections permitted per second per daemon. After this many connections are accepted, further connections will be delayed. It will protect agains denial-of-service attacks and spammers opening hundreds of connections to your server.

Allow incoming mail relay

Add the domains you wish to serve to /etc/mail/local-host-names:

targetdomain1.com
targetdomain2.com
targetdomain3.com

Domain-based routing: the mailertable

This is a feature that allows you to route mail to certain domains via given mail gateways.

The /etc/mail/mailertable could look like this (please edit before deploying):

targetdomain1.com          smtp:targethost1.example.com
targetdomain2.com          smtp:targethost2.test.net
targetdomain3.com          smtp:[192.168.0.1]

Allow incoming relay: the access

The mail domains your server is allowed to relay from outside (the Internet) must be listed in the /etc/mail/access

To:targetdomain1.com  RELAY
To:targetdomain2.com  RELAY
To:targetdomain3.com  RELAY

This feature needs the access_db feature to be enabled in sendmail.mc.

Don't put these domains in the /etc/mail/local-hosts file (you should put there only the mail domains your host will receive itself).

Allow outgoing relay (optional)

This part is optional but useful. If you want hosts in your internal network to be able to use the server as an outgoing relay, add the hosts to the file /etc/mail/access:

192.168.0.    RELAY
192.168.1.1   RELAY
192.168.1.2   RELAY

Other configuration files

Also, create some files sendmail wants to see in /etc/mail:

touch /etc/mail/domaintable
touch /etc/mail/virtusertable
touch /etc/mail/local-host-names

Make the configuration files readable by everybody:

chmod 644 /etc/mail/*

Make sure your /etc/syslog.conf includes a line like:

mail.debug                      /var/log/maillog

That will put mail related entries into their own log file.

Updating configuration with make

To keep configuration files and Sendmail database files up to date, I usually create a Makefile which includes the rules to compile all changed configuration files. This is what I used here. The contents of /etc/mail/Makefile look like this:

all: access.db access.dir access.pag \
     aliases.db aliases.dir aliases.pag \
     domaintable.db domaintable.dir domaintable.pag \
     mailertable.db mailertable.dir mailertable.pag \
     virtusertable.db virtusertable.dir virtusertable.pag \
     sendmail.cf

access.db: access
        makemap hash /etc/mail/access </etc/mail/access

access.dir access.pag: access
        makemap dbm /etc/mail/access </etc/mail/access

aliases.db: aliases
        makemap hash /etc/mail/aliases </etc/mail/aliases

aliases.dir aliases.pag: aliases
        makemap dbm /etc/mail/aliases </etc/mail/aliases

domaintable.db: domaintable
        makemap hash /etc/mail/domaintable </etc/mail/domaintable

domaintable.dir domaintable.pag: domaintable
        makemap dbm /etc/mail/domaintable </etc/mail/domaintable

mailertable.db: mailertable
        makemap hash /etc/mail/mailertable </etc/mail/mailertable

mailertable.dir mailertable.pag: mailertable
        makemap dbm /etc/mail/mailertable </etc/mail/mailertable

virtusertable.db: virtusertable
        makemap hash /etc/mail/virtusertable </etc/mail/virtusertable

virtusertable.dir virtusertable.pag: virtusertable
        makemap hash /etc/mail/virtusertable </etc/mail/virtusertable

sendmail.cf: sendmail.mc
        cd /usr/samples/tcpip/sendmail/m4; \
        m4 cf.m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

That way, after you make changes to configuration files, just go to /etc/mail and run make. That will update all configuration files as needed. Give the sendmail process a kill -HUP after that to make it read its configuration again.

cd /etc/mail
make
kill -HUP

Running Sendmail

Start command:

startsrc -s sendmail -a "-bd -q30m"

Stop command:

stopsrc -s sendmail

Update configuration (see section above):

cd /etc/mail
make
kill -HUP

See the queue:

sendmail -bp

Watch your mail log:

tail -f /var/log/maillog

Watch while Sendmail processes the mail queue:

sendmail -q -v

Routing your mail: the DNS records

When your new relay server is up and running, and verified to work correctly, it is time to update the DNS records of the domains to route mail through your new server(s).

targetdomain1.com. IN MX 10 mail1.koo.fi.
targetdomain2.com. IN MX 10 mail2.koo.fi.

The second part is needed only if you configured two servers. The number "10" there is priority. By giving both servers the same priority, incoming mail will be split somewhat equally between them.

Also make sure your own domain has A records for the mail server:

mail1.koo.fi. IN A 192.168.1.1
mail2.koo.fi. IN A 192.168.1.2

And third, it is also crucial to add your servers to your IP network's reverse lookup DNS zone. In this example case it would be the 1.168.192.in-addr.arpa domain:

1.1.168.192.in-addr.arpa. IN PTR mail1.koo.fi.
2.1.168.192.in-addr.arpa. In PTR mail2.koo.fi.

A reverse mapping is needed, because a lot of Internet's mail servers are configured to check for its existence. This, once again, to make life harder for spammers.

NIC bonding with Ubuntu

2008-08-02T21:00:00+03:00

Network interfaces can be bonded to provide fault-tolerant operation. Here's how to do it in Ubuntu. I will assume the interfaces to be bonded are eth0 and eth1.

First, install the ifenslave package. The ifenslave tool will be used to actually bond the interfaces.

apt-get install ifenslave

Create file /etc/modprobe.d/bonding with the following contents:

alias bond0 bonding
options bonding mode=0 miimon=100

Load the bonding module:

modprobe bonding

Add a bonded interface into /etc/network/interfaces:

auto bond0
iface bond0 inet static
  address 192.168.0.1
  gateway 192.168.0.254
  netmask 255.255.255.0
  pre-up modprobe bonding
  up ifenslave bond0 eth0 eth1
  pre-down ifenslave bond0 -d eth0 eth1
  post-down rmmod bonding

Restart networking:

/etc/init.d/networking restart

That's it! The bonded interface should come up automatically after a reboot as well.

Windows Server Time with NTP

2008-08-02T13:34:00+03:00

Here's how to configure a Windows domain controller to act as an NTP client and server for your network. You may then sync all your hosts, Windows or other, to that server. To achieve this, configure one (or more) of your domain controllers to retrieve time from the atomic clocks of the Internet. Rest of you servers should follow suit and sync their time to this domain controller after a little while.

You must allow the NTP protocol to pass through your firewall. This can be achieved by allowing tcp/123 and udp/123 traffic.

To retrieve time from the Internet and serve it to computers on your network, edit or add the following registry keys:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config]
"AnnounceFlags"=dword:00000005

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters]
"NtpServer"="0.europe.pool.ntp.org,0x1 1.europe.pool.ntp.org,0x1 2.europe.pool.ntp.org,0x1"
"Type"="NTP"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpClient]
"SpecialPollInterval"=dword:00000e10

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpServer]
"Enabled"=dword:00000001

That configuration will effectively make your server both an NTP client and an NTP server. SpecialPollInterval is the polling interval in seconds. 0xe10 = 3600 seconds. The NtpServer string is a space-separated list of time providers, each appended with the string ",0x1". I use ntp.org European time providers.

Restart the W32Time service to make the configuration current:

net stop w32time
net start w32time

To force resynchronization of time:

w32tm /resync

To monitor the time of your domain controllers:

w32tm /monitor

To force resynchronization of time on some other server:

w32tm /resync /computer:REMOTESERVER

You can now configure your non-windows boxes to sync time to the server using NTP. Windows machines which are AD domain members should sync automatically without further configuration.

Links:

http://support.microsoft.com/kb/816042

http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/security/ws03mngd/26_s3wts.mspx

Adding the First Windows 2008 DC into Active Directory

2008-06-25T14:49:00+03:00

Make a backup copy of your AD before you go any further.

Install your new server, and join it to the domain as a member server. Before you can run dcpromo on the new 2008 server, you must run adprep on your schema master, to prepare the Active Directory schema to support Windows 2008 domain controllers. The installation DVD contains a directory called sourcesadprep. Go there and run:

D:sourcesadprep>adprep /forestprep
D:sourcesadprep>adprep /domainprep
D:sourcesadprep>adprep /rodcprep

After that you may run dcpromo to promote the server as a domain controller.

Fetching information from Active Directory using Python

2008-06-19T10:17:00+03:00

Here are two simple scripts written in Python to fetch information about users from Active Directory. The AD schema has been augmented with the Microsoft Services For Unix schema, which will allow to map Unix uids to Windows user accounts.

All you need to do is to fill in your own domain controller, AD distinguished name (user account) and the password for it in /etc/ad.secret. You shoud use a less privileged account than Administrator for security. Also remember to set the permissions for ad.secret so that only privileged users have access.

This one will fetch the real name of a user when given a Unix uid:

#!/usr/bin/python

import sys
import ldap

if len(sys.argv) < 2:
  print "usage: getrealname <username>"
  sys.exit()

Server = "dc1.koo.fi"
DN = "cn=Administrator,cn=Users,dc=koo,dc=fi"
Secret = file("/etc/ad.secret").readline().strip()
Base = "dc=koo,dc=fi"
Scope = ldap.SCOPE_SUBTREE
Filter = "(&(objectClass=user)(msSFU30Name="+sys.argv[1]+"))"
Attrs = ["displayName", "msSFU30Name"]
l = ldap.open(Server)
l.simple_bind(DN, Secret)
r = l.search(Base, Scope, Filter, Attrs)
Type,user = l.result(r,60)
Name,Attrs = user[0]
if hasattr(Attrs, 'has_key') and Attrs.has_key('displayName'):
  displayName = Attrs['displayName'][0]
  print displayName

sys.exit()

And this one will do the same for e-mail:

#!/usr/bin/python

import sys
import ldap

if len(sys.argv) < 2:
  print "usage: getemail <username>"
  sys.exit()

Server = "dc1.koo.fi"
DN = "cn=Administrator,cn=Users,dc=koo,dc=fi"
Secret = file("/etc/ad.secret").readline().strip()
Base = "dc=koo,dc=fi"
Scope = ldap.SCOPE_SUBTREE
Filter = "(&(objectClass=user)(msSFU30Name="+sys.argv[1]+"))"
Attrs = ["mail", "msSFU30Name"]
l = ldap.open(Server)
l.simple_bind(DN, Secret)
r = l.search(Base, Scope, Filter, Attrs)
Type,user = l.result(r,60)
Name,Attrs = user[0]
if hasattr(Attrs, 'has_key') and Attrs.has_key('mail'):
  mail = Attrs['mail'][0]
  print mail

sys.exit()

Apache HTTP authentication to Active Directory with Kerberos

2008-06-18T19:52:00+03:00

First, create a user account for your Apache in the Active Directory. Let's assume the AD Kerberos realm is KOO.FI, and the user name we have created is "apache". Also create a computer account, let's call that "apachesrv".

Next, create two keytab files on the Windows server. One host keytab file and one service keytab file (long lines have been split):

C:>ktpass -princ HOST/www.koo.fi@KOO.FI -mapuser apachesrv@KOO.FI
-crypto DES-CBC-MD5 -DesOnly -pass XXXCHOOSEXAXSECRETXWORDXXX
-ptype KRB5_NT_SRV_HST -out krb5.keytab

C:>ktpass -princ HTTP/www.koo.fi@KOO.FI -mapuser apache@KOO.FI
-pass XXXSECRETXXX -out keytab.HTTP

Make sure that the principal name you are using (HTTP/your.server.com) has the actual domain name that is being requested from Apache by the web browser. If they differ, you will end up having error messages saying "failed to verify krb5 credentials: Server not found in Kerberos database" in you Apache error log.

You should now have binary files called krb5.keytab and keytab.HTTP in your current directory. Copy those files over to your Apache server into /etc.

Edit /etc/krb5.conf:

[libdefaults]
        default_realm = KOO.FI
        default_keytab_file = /etc/krb5.keytab
        dns_lookup_realm = true
        dns_lookup_kdc = true

[realms]
        KOO.FI = {
                kdc = dc1.koo.fi
                kdc = dc2.koo.fi
                admin_server = dc1.koo.fi
        }
[domain_realm]
        .koo.fi = KOO.FI
        koo.fi = KOO.FI

Test that your authentication works:

root@apachesrv:/etc# kinit HOST/www.koo.fi
Password for HOSTt/www.koo.fi@KOO.FI:

Enter the secret string you used earlier to create the machine account. If everything went correctly, you should be able to list the ticket:

root@apachesrv:/etc# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HOST/www.koo.fi@KOO.FI

Valid starting     Expires            Service principal
06/11/08 15:26:55  06/12/08 01:25:16  krbtgt/KOO.FI@KOO.FI
    renew until 06/12/08 15:26:55

Kerberos 4 ticket cache: /tmp/tkt0
klist: You have no tickets cached

Lastly, let's configure Apache. My Apache server happened to be an Ubuntu box with Apache 2.2 installed. The Apache module mod_auth_kerb will take care of the authentication, so let's install that:

root@apachesrv:/etc# aptitude install libapache2-mod-auth-kerb

Add a directory directive in your Apache configuration file:

<Directory /var/www/www.koo.fi/protected>
  AuthType Kerberos
  KrbMethodNegotiate on
  KrbMethodK5Passwd on
  KrbAuthoritative on
  KrbAuthRealms KOO.FI
  KrbVerifyKDC on
  KrbServiceName HTTP
  Krb5Keytab /etc/keytab.HTTP
  KrbSaveCredentials off
  AuthName "This url is protected. Keep your unauthorized hands off!"
  Require Valid-user
</Directory>

Reload the changes to Apache, and you're all set!

root@apachesrv:/etc# /etc/init.d/apache2 force-reload

Some links:

http://sl.mvps.org/docs/LinuxApacheKerberosAD.htm
http://blog.scottlowe.org/2006/08/08/linux-active-directory-and-windows-server-2003-r2-revisited/
http://modauthkerb.sourceforge.net/configure.html

HP Array Configuration and Diagnostic Utilities on Linux

2008-06-08T14:52:00+03:00

Getting the HP Array Configuration Utility (ACU) and the Array Diagnostic Utility (ADU) for Linux to work was non-trivial. It does not seem to be supported anymore, but I managed to get it working on CentOS 5 running on an HP ProLiant DL185 G5.

First, install compatibility packages:

sudo yum install compat-libstdc++-33.i386 compat-libstdc++-33.x86_64 \
                 compat-libstdc++-296.i386

Then, download and install the packages HP provides:

wget -nd ftp://ftp.hp.com/pub/products/servers/supportsoftware/linux/hpacucli-7.70-12.linux.rpm
wget -nd ftp://ftp.hp.com/pub/products/servers/supportsoftware/linux/hpadu-7.70-12.linux.rpm
wget -nd ftp://ftp.hp.com/pub/products/servers/supportsoftware/linux/hpsmh-2.1.7-168.linux.x86_64.rpm
sudo rpm -i --nodeps hpacucli-7.70-12.linux.rpm
sudo rpm -i --force hpadu-7.70-12.linux.rpm hpsmh-2.1.7-168.linux.x86_64.rpm

The Array Configuration command line utility can be started with the command:

sudo -i hpacucli

The utility itself is quite user-friendly, and the help seems to be good. Just type "help" and you will find commands to show and modify your arrays. Some examples:

=> ctrl all show status

Smart Array E200
   Controller Status: OK
   Cache Status: OK
   Battery Status: OK

=> ctrl all show config

Smart Array E200              (sn: PA6C90L9SV4152)

   array A (SATA, Unused Space: 0 MB)

      logicaldrive 1 (1.4 TB, RAID 5, OK)

      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 500.1 GB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 500.1 GB, OK)
      physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 500.1 GB, OK)
      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA, 500.1 GB, OK)

The utility also understands commands directly from the command line:

$ sudo -i hpacucli ctrl serialnumber=PA6C90L9SV4152 array A show

Smart Array E200

   Array: A
      Interface Type: SATA
      Unused Space: 0 MB
      Status: OK

To create a diagnostic report with the Array Diagnostic Utility, type:

sudo /usr/sbin/hpaducli -f hpadu.report

That will create a detailed diagnostic report into the file "hpadu.report". You can use that report to check what has failed in case you controller or array status is not OK.

I created a little script called "array_check" that can be added as a cron job to check that the array is healthy:

#!/bin/sh

/usr/sbin/hpacucli ctrl serialnumber=PA6C90L9SV4152 array A show status|grep -q "array A: OK"

if test "$?" != "0"
then
  echo Array A not ok!
  hpacucli ctrl serialnumber=PA6C90L9SV4152 array A show
  hpacucli ctrl serialnumber=PA6C90L9SV4152 show
  echo ------------------------------------------------------------
  echo
  /usr/sbin/hpaducli -f /tmp/hpadu.report
  cat /tmp/hpadu.report
fi

Just add the script as a cron job and configure your machine to e-mail you the output of cron jobs, and you will get an e-mail if the status of the array changes.

Lenovo ThinkPad X300 Ubuntu 8.04 Installation Notes

2008-06-05T21:12:00+03:00

First, let me tell you some first impressions about the machine. The keyboard is very good. It feels even a bit better than the one in my old T60. The display is very bright and sharp, but viewing angles could be better. WLAN worked right out of the box, as did the webcam.

The solid state drive is incredibly fast. I will never switch back to a hard disk after experiencing an SSD. Everything loads up in an instance. OpenOffice starts in about 5 seconds, which is very good compared to my T60. And the machine boots up and shuts down really fast (I haven't timed those operations, though).

There are also a couple of annoyances which I hope will soon be fixed. Under ubuntu, the fan keeps running at nearly 6000 RPM and makes a lot of noise. It only stops when you let the machine settle a while with the display off, and starts again almost immediately you start using it again. This is the most annoying thing I can think of. Hopefully somebody finds a fix for this one. The temperatures of various components remain reasonably low in my opinion.The CPU runs at about 40

Limiting the bandwidth of incoming traffic

2008-04-23T15:48:00+03:00

A backup server was saturating the DSL links of remote offices every time the backups were running. To prevent this, I had to limit the incoming bandwidth of the TCP-connections that were used to back up the remote hosts, but not touch the ones that were used to connect to the servers in the local network. Here's how to do it.

The limiting must be done by attaching a queuing discipline on the ingress side of the interface the traffic is coming in from. After that you can attach filters that filter traffic from specified hosts or subnets with a given bandwidth.

Step number one is to attach the ingress qdisc:

root@srv:/# tc qdisc add dev eth0 handle ffff: ingress

Step number two is to add one or more filters that police the bandwidth:

root@srv:/# tc filter add dev eth0 parent ffff: protocol ip prio 50 \
            u32 match ip src 192.168.123.123 \
            police rate 64kbit burst 10k drop flowid :1
root@srv:/# tc filter add dev eth0 parent ffff: protocol ip prio 50 \
            u32 match ip src 192.168.124.0/24 \
            police rate 128kbit burst 10k drop flowid :1

Step number three is to make your new qdisc and filters to load each time you reboot your server. On Ubuntu, this can be achieved by adding a script to the /etc/network/if-up.d directory. Scripts in that directory will be called whenever a network interface comes up.

Let's add a file called /etc/network/if-up.d/tc with the following contents:

#!/bin/sh
#
# Network traffic control settings
#

/sbin/tc qdisc add dev eth0 handle ffff: ingress
/sbin/tc filter add dev eth0 parent ffff: protocol ip prio 50 \
         u32 match ip src 192.168.123.123 \
         police rate 64kbit burst 10k drop flowid :1
/sbin/tc filter add dev eth0 parent ffff: protocol ip prio 50 \
         u32 match ip src 192.168.124.0/24 \
         police rate 128kbit burst 10k drop flowid :1

MediaWiki Image Links

2008-04-14T16:02:00+03:00

Here's a description how to do it without any extensions:

http://distributedresearch.net/blog/2008/01/21/how-to-make-image-links-in-mediawiki

This extension will allow you to use more comfortable markup:

` http://www.mediawiki.org/wiki/Extension:ImageLink <http://www.mediawiki.org/wiki/Extension:ImageLink>`__

Mounting Windows filesystems from AIX 5.3

2008-02-18T14:52:00+02:00

First, you must install fileset bos.cifs_fs.rte, and optionally bos.cifs_fs.smit for the Smitty interface, from the installation DVD.

To invoke the Smitty interface, run:

root# smitty cifs_fs

The Smitty interface will enable you to do almost anything you need with the CIFS filesystem.

For people aligned the command line way, mounting a share is a two-phase process. First, credentials must be added to the /etc/cifs_fs/cifscred file. After that, you can mount a remote CIFS file system using the credentials specified. You could do the whole thing from the command line, but this is the preferred method in my opinion.

SMBFS can store server/user/password credentials in the /etc/cifs_fs/cifscred file to allow automatic retrieval of passwords when mounting SMBFS. Credentials can be added, changed, and removed from this file with the mkcifscred, chcifscred, and rmcifscred commands.

To add a credential:

root# mkcifscred -h <server> -u <user> -p <pass>

Use the mkcifsmnt, chcifsmnt, rmcifsmnt, and lscifsmnt commands to add, change, remove, and list, respectively, cifs stanzas in /etc/filesystems.

To mount a file system:

root# mkdir -p /mount/point
root# mkcifsmnt -f /mount/point -h <server> -d <share> -c <user> -w <workgroup/domain>

That will add a new stanza to /etc/filesystems:

/mount/point:
        dev             = <share>
        vfs             = cifs
        nodename        = <server>/&lt:user>
        mount           = false
        options         = wrkgrp=<workgroup/domain>
        account         = false

Automatical mounting during startup can be specified with the -A option to mkcifsmnt. You can give your mounted share a different set of permissions and owners with the -u <uid>, -g <gid> and the -x <mode> options.

Sources of information:

Sqlite3 for ARMEL

2008-02-10T01:47:00+02:00

Here's a statically compiled version of Sqlite 3.5.6 command line interface on ARMEL with Readline support. It runs nicely on Nokia Internet Tablets with the OS 2008 (libreadline4 must be installed):

sqlite3-armel.gz (download no longer available)

It was compiled with the following settings:

./configure --disable-shared --enable-readline --enable-dynamic-extensions …

MS Exchange 2007 upgrade memo

2008-01-14T10:20:00+02:00

This short list should apply to a situation where an organization with one domain and a simple Exchange environment wants to upgrade from Exchange 2003 to 2007. It assumes everything will be installed on a single server with a fresh installation of Windows 2003 R2, joined as a member server into the domain.

Software requirements:

Microsoft Internet Information Server
.NET Framework 2.0 SP1
Windows PowerShell

Active Directory requirements:

Forest Functional Level must be at least 2003
Domain Function Level must be at least 2003
Schema Master must be a Windows 2003 SP1 or newer
Operations Master must be a Windows 2003 SP1 or newer
Echange Server operation mode must be Native

To prepare Active Directory, the following operations must be performed with the Setup.exe that is included in the Exchange Server installation media. Just run them from the command line.

setup /pl -- short for /PrepareLegacyExchangePermissions
setup /ps -- short for /PrepareSchema
setup /p -- short for /PrepareAD
setup /pd -- short for /PrepareDomain

After that, you are ready to run setup and start migrating mailboxes.

The default installation installs Outlook Web Access into your Default Web Site under /owa. A quick way to forward incoming http and https request to the server root directly into OWA login page can be done this way:

Disable "Require secure channel (SSL)" from the Default Web Site properties
Enable "Require secure channel (SSL)" from the /owa folder properties
Add a "Default.aspx" in the server root, with the following content:
```
<%
response.redirect("https://your.server.com/owa")
%>
```

Scratchbox installation under 32-bit chroot on 64-bit Ubuntu

2008-01-05T17:10:00+02:00

I wanted to try out the Maemo SDK for Nokia 770, N800 and N810 devices (for some reason I happen to own one of every generation), but found out that there are no prebuilt packages for the 64-bit environment. The quick (?) remedy for this is the chroot jail, because a guest i386 environment can pretty easily be bootstrapped inside a 64-bit one. Here are the step-by-step instructions for doing it.

Setting up the chroot jail for Scratchbox

The first thing to do is setting up a chroot environment with 32-bit binaries. This can be achieved nicely with schroot and debootstrap, which can be found in the Debian Universe. Debootstrap will download and install a basic Debian system into a directory. In this case, we will install Ubuntu Gutsy i386 under /ubuntu32:

root@T60:/# apt-get install schroot debootstrap
root@T60:/# mkdir /ubuntu32
root@T60:/# debootstrap --arch i386 gutsy /ubuntu32
I: Retrieving Release
I: Retrieving Packages
I: Validating Packages
.
.
.
root@T60:/# chroot /ubuntu32

That last command will actually chroot you into the directory /ubuntu32. What that means in practise is that your new root directory will be in fact /ubuntu32, and your new shell will be restricted under that directory. In effect, what you see as the root directory / will in reality reside under /ubuntu32. There is no easy way to see whether you're inside a jail or not, and the purpose of this is of course to trick programs into thinking they are running on another system. To escape the chroot jail, just say exit.

The newly bootstrapped system is very bare. Indeed there is no locale support until you install it. That should probably be the first thing to do. My locale is Finnish, so I will install the language-pack-fi package. After that, let's install the Nano editor, and edit our Apt sources:

root@chroot:/# apt-get install language-pack-fi
root@chroot:/# apt-get install nano
root@chroot:/# nano -w /etc/apt/sources.list

I make the sources.list look like this:

deb http://archive.ubuntu.com/ubuntu gutsy main restricted universe multiverse

A couple more things to do before our chroot is ready. We must set up user accounts and mounts. If you are happy running as the root user, there is no need to do anything, but I wanted to add an account for myself. The easiest way is to copy all necessary information from your existing /etc outside the chroot:

root@T60:/etc# cp passwd shadow group gshadow sudoers hosts resolv.conf /ubuntu32/etc/

That way your password, groups and all other settings are preserved without problems. You can add that command to cron if you want them synchronized automatically.

Next, you need to mount a couple of filesystems to make things go smoothly. Such are /proc, /sys and /dev, and also possibly /home and /tmp. The first three are needed for correct operation of many programs, while the latter two will ease your use of the chroot environment.

The create the mounts, add these lines to the /etc/fstab OUTSIDE the chroot (your normal fstab):

# chroot binds:

/home  /ubuntu32/home   none   bind     0 0
/tmp   /ubuntu32/tmp    none   bind     0 0
/dev   /ubuntu32/dev    none   bind     0 0
proc   /ubuntu32/proc   proc   defaults 0 0
sysfs  /ubuntu32/sys    sysfs  defaults 0 0

I mounted them right away with the mount -a command.

The last thing to do is make the chroot accessible for the general user. This is where schroot comes into play. It's purpose is to make normal users able to chroot themselves into jails predefined by root. Let's add a chroot jail into /etc/schroot/scroot.conf:

[ubuntu32]
description=Gutsy Gibbon i386
location=/ubuntu32
users=mkortela
aliases=scratchbox,default

Now, user mkortela is able to chroot into our new environment:

mkortela@T60:~$ schroot ubuntu32
I: [ubuntu32 chroot] Running login shell: '/bin/bash'
mkortela@T60:~$

Now we are in the chroot jail as ourselves, and able to sudo as we were outside the jail. Also, the user's home directory is right where it is supposed to be. To remember we are inside the chroot, it may be a good idea to change the prompt a little bit:

mkortela@T60:~$ export PS1='${debian_chroot:+($debian_chroot)}u@chroot:w$ '
mkortela@chroot:~$

Scratchbox installation inside the chroot jail

The Scratchbox installation will go pretty much like described in the INSTALL.txt. There are a couple of extra tweaks, though. First, lets fetch the installation scripts, and run them. First the Scratchbox installation, then the SDK installation. The installation scripts will have to be cheated with the linux32 command:

mkortela@chroot:~$ sudo apt-get install wget
mkortela@chroot:~$ wget http://tablets-dev.nokia.com/4.0/maemo-scratchbox-install_4.0.sh
mkortela@chroot:~$ wget http://tablets-dev.nokia.com/4.0/maemo-sdk-install_4.0.sh
mkortela@chroot:~$ chmod a+x maemo-scratchbox-install_4.0.sh
mkortela@chroot:~$ chmod a+x maemo-sdk-install_4.0.sh
mkortela@chroot:~$ linux32 sudo ./maemo-scratchbox-install_4.0.sh
.
.
.
Installation was successful!
----------------------------

You now have Scratchbox 1.0.8 'apophis' release installed.

Scratchbox cannot be run as user root. Instead, use your normal login
user account. Add additional scratchbox users and sandboxes with the
following command (outside scratchbox with root permissions):

        # /scratchbox/sbin/sbox_adduser USER yes

Running this command will create sandbox environment for that user and
add user to the 'sbox' scratchbox user group.
You will need to start a new login terminal after being added to the
'sbox' group for group membership to be effective.

Login to scratchbox session using the following command (as user):

        $ /scratchbox/login

Refer to scratchbox.org documentation for more information re scratchbox:
http://scratchbox.org/documentation/user/scratchbox-1.0/

Let's do what they ask:

mkortela@chroot:~$ sudo /scratchbox/sbin/sbox_adduser mkortela yes
Adding user `mkortela' to group `sbox' ...
Done.
Scratchbox user account for user mkortela added

And let's create the group outside the chroot and add ourselves there as well:

root@T60:/# groupadd -g 1001 sbox
root@T60:/# usermod -a -G sbox mkortela

And now, the SDK installation:

mkortela@chroot:~$ linux32 ./maemo-sdk-install_4.0.sh

I installed with the default options, although I decided to download the non-free Nokia packages as well.

Installing and running Xephyr

Xephyr can be installed directly from the universe. We will install it outside the chroot environment, because that's where all our X libraries and fonts are:

root@T60:/# apt-get install xserver-xephyr

Let's check that it runs as a regular user:

mkortela@T60:~$ Xephyr :2 -host-cursor -screen 800x480x16 -dpi 96 -ac -extension Composite

You should get a Window with a simple X session (just the default boring wallpaper and nothing else). To start the Maemo default programs in it, open another shell window, go inside the chroot, and run:

mkortela@chroot:~$ sudo /etc/init.d/scratchbox-core start
mkortela@chroot:~$ sb-conf select CHINOOK_X86
mkortela@chroot:~$ scratchbox
[sbox-CHINOOK_X86: ~] > export DISPLAY=:2
[sbox-CHINOOK_X86: ~] > af-sb-init.sh start

You should be able to see a very basic Maemo session, similar to the one shown in the Maemo 4.0 Tutorial. That document is also a good place to continue from here on.

SUID script requirements on AIX 5L and ksh93

2007-11-30T16:45:00+02:00

Requirements for a suid script include:

#! directing the KornShell be used
Executable by user, group, and other
No read permission
Add suid permission by chmod u+s on the file

Add a -p option to #! to increase security to force a separate process if one is not normally done.

Example:

#! /usr/bin/ksh -p

Source of this information:

http://www.ibm.com/developerworks/aix/library/au-kornshell93.html#n

Enabling security on an HP ProCurve 4200 series switch

2007-11-03T15:34:00+02:00

I had a chance to configure an HP ProCurve 4208vl switch the other day. The first impression was that the command line interface is heavily influenced by, if not directly copied from, the Cisco IOS command line interface. So if you have experience with IOS, you will probably feel almost at home on an HP switch. There are some differences, though.

The first thing I wanted to do was to enable ssh access and authentication, and disable telnet. Here's a quick howto.

Connect to the switch using the console cable or telnet.

First thing to do is to enter the configuration mode and generate a key for ssh. Only after the key has been generated is it possible to enable ssh:

ProCurve Switch 4208vl# configure
ProCurve Switch 4208vl(config)# crypto key generate ssh
depleted, this could take up to a minute.
ProCurve Switch 4208vl(config)# ip ssh
ProCurve Switch 4208vl(config)# ip ssh filetransfer
ProCurve Switch 4208vl(config)# end

That is not enough, however. You must set the operator and manager passwords to actually authenticate to the switch.

ProCurve Switch 4208vl# configure
ProCurve Switch 4208vl(config)# password manager
New password for Manager:
Please retype new password for Manager:
ProCurve Switch 4208vl(config)# password operator
New password for Operator:
Please retype new password for Operator:
ProCurve Switch 4208vl(config)# end

After the above changes, the web interface will also require a password. For some reason, you must leave the username field empty and input either the manager or the operator password in the password field.

To disable the telnet server:

ProCurve Switch 4208vl# configure
ProCurve Switch 4208vl(config)# no telnet-server
ProCurve Switch 4208vl(config)# end

To create a key and a self-signed certificate for SSL web access:

ProCurve Switch 4208vl(config)# crypto key generate cert 1024
Installing new RSA key.  If the key/entropy cache is
depleted, this could take up to a minute.
ProCurve Switch 4208vl(config)# crypto host-cert generate self-signed 11/01/2007 11/01/2017 sw1.koo.fi _ Techelp Helsinki _ fi
ProCurve Switch 4208vl(config)# web-management ssl
ProCurve Switch 4208vl(config)# aaa authentication web login local
ProCurve Switch 4208vl(config)# end

Write your configuration changes:

ProCurve Switch 4208vl# write memory

Installing MySQL 5 on IBM AIX 5.3

2007-11-02T12:40:00+02:00

The IBM AIX Software Toolbox download page includes a package for MySQL 3.23, but that was a little bit too aged for my purposes. Fortunately MySQL distributes binaries for IBM AIX here:

http://dev.mysql.com/downloads/mysql/5.0.html#aix

That package will work out of the box with 5.3 as well. Download it, and unpack it to /usr/local:

root@server# cat mysql-5.0.45-aix5.2-powerpc-64bit.tar.gz|gunzip|tar xvC /usr/local
root@server# ln -s /usr/local/mysql-5.0.45-aix5.2-powerpc-64bit /usr/local/mysql

There is a file named INSTALL-BINARY which includes these instructions as well as some more information, but here's a quick overview of the things you need to take care of.

Create a user and group for privilege separation:

root@server# groupadd mysql
root@server# useradd -g mysql mysql
root@server# chown -R mysql:mysql /usr/local/mysql-5.0.45-aix5.2-powerpc-64bit

Create the MySQL data directory and initialize the grant tables:

root@server# cd /usr/local/mysql
root@server# ./scripts/mysql_install_db --user=mysql

Only the data directory need to be owned by the mysql user, so for security reasons, we will give all other files back to root:

root@server# chown -R root /usr/local/mysql-5.0.45-aix5.2-powerpc-64bit
root@server# chown -R mysql /usr/local/mysql-5.0.45-aix5.2-powerpc-64bit/data

Start it up:

root@server# /usr/local/mysql/bin/mysqld_safe --user=mysql &

IBM Websphere Application Server 5.1 init script for Linux

2007-11-02T12:13:00+02:00

WAS does not seem to install any default init scripts, so I created one. You can configure it to start and stop multiple application servers by listing them all in the APPSERVERS variable (separated by spaces).

#!/bin/sh
# chkconfig: 35 99 10
# description: Starts and stops Websphere Application Server

. /lib/lsb/init-functions

# Please edit these variables to match your environment. You may list one or more
# application servers separated by spaces in the APPSERVERS variable.

OWNER=wasadmin
USERNAME=wasuser
PASSWORD=secret
APPSERVERS="Appserver1"

WASDIR=/opt/WebSphere/AppServer
DMGRDIR=/opt/WebSphere/DeploymentManager
LOCKDIR=/var/lock/subsys

DMGRLOCK="$LOCKDIR/WAS_deploymentmgr"
NODEAGENTLOCK="$LOCKDIR/WAS_nodeagent"

# Don't edit beyond this point.

start_dmgr() {
  MSG="Starting WAS Deployment Manager"
  [[ ! -e $DMGRLOCK ]] \
      && su - $OWNER -c "$DMGRDIR/bin/startManager.sh -user $USERNAME -password $PASSWORD" \
      && touch $DMGRLOCK \
      && log_success_msg $MSG \
      || log_failure_msg $MSG
}

stop_dmgr() {
  MSG="Stopping WAS Deployment Manager"
  [[ -e $DMGRLOCK ]] \
    && [[ -e "$DMGRDIR/bin/stopManager.sh" ]] \
      && su - $OWNER -c "$DMGRDIR/bin/stopManager.sh -user $USERNAME -password $PASSWORD" \
      && rm -f $DMGRLOCK \
      && log_success_msg $MSG \
      || log_failure_msg $MSG
}

start_nodeagent() {
  MSG="Starting WAS Node Agent"
  [[ ! -e $NODEAGENTLOCK ]] \
    && su - $OWNER -c "$WASDIR/bin/startNode.sh -user $USERNAME -password $PASSWORD" \
    && touch $NODEAGENTLOCK \
    && log_success_msg $MSG \
    || log_failure_msg $MSG
}

stop_nodeagent() {
  MSG="Starting WAS Node Agent"
  [[ -e $NODEAGENTLOCK ]] \
    && su - $OWNER -c "$WASDIR/bin/stopNode.sh -user $USERNAME -password $PASSWORD" \
    && rm -f $NODEAGENTLOCK \
    && log_success_msg $MSG \
    || log_failure_msg $MSG
}

start_appservers() {
  for WAS in $APPSERVERS
  do
    MSG="Starting WAS: $WAS"
    LOCK="$LOCKDIR/WAS_$WAS"
    [[ ! -e $LOCK ]] \
      && su - $OWNER -c "$WASDIR/bin/startServer.sh $WAS -user $USERNAME -password $PASSWORD" \
      && touch $LOCK \
      && log_success_msg $MSG \
      || log_failure_msg $MSG
  done
}

stop_appservers() {
  for WAS in $APPSERVERS
  do
    MSG="Stopping WAS: $WAS"
    LOCK="$LOCKDIR/WAS_$WAS"
    [[ -e $LOCK ]] \
      && su - $OWNER -c "$WASDIR/bin/stopServer.sh $WAS -user $USERNAME -password $PASSWORD" \
      && rm -f $LOCK \
      && log_success_msg $MSG \
      || log_failure_msg $MSG
  done
}


case "$1" in
  'start')   [[ -e "$DMGRDIR/bin/startManager.sh" ]] && start_dmgr
             start_nodeagent
             start_appservers ;;
  'stop')    stop_appservers
             stop_nodeagent
             [[ -e "$DMGRDIR/bin/startManager.sh" ]] && stop_dmgr ;;
  'restart') stop_appservers
             stop_nodeagent
             [[ -e "$DMGRDIR/bin/startManager.sh" ]] && stop_dmgr
             sleep 5
             [[ -e "$DMGRDIR/bin/startManager.sh" ]] && start_dmgr
             start_nodeagent
             start_appservers ;;
  'status')  [[ -e $DMGRLOCK ]] \
               && echo WAS Deployment Manager is running (lockfile $DMGRLOCK) \
               || echo WAS Deployment Manager is stopped (no lockfile)
             [[ -e $NODEAGENTLOCK ]]
               && echo WAS Node Agent is running (lockfile $NODEAGENTLOCK) \
               || echo WAS Node Agent is stopped (no lockfile)
             for WAS in $APPSERVERS
             do
               LOCK="$LOCKDIR/WAS_$WAS"
               [[ -e $LOCK ]] \
                 && echo WAS $WAS is running (lockfile $LOCK) \
                 || echo WAS $WAS is stopped (no lockfile)
             done
             $WASDIR/bin/serverStatus.sh -all -user $USERNAME -password $PASSWORD
             ;;
  *)         echo "Usage: was (start|stop|restart|status)"
             exit 1 ;;
esac

WPA-PSK authentication with Cisco IOS

2007-10-26T20:50:00+03:00

When my Linux firewall box died a couple of months ago, I finally decided to by a Cisco router for my Internet connection. Before the Linux box I had an OpenBSD firewall, and I decided it was time to learn yet another platform.

The box is a Cisco 877W which has one ADSL interface, a four-port ethernet switch, and an 802.11g wireless NIC. My first impressions have been very positive. I have for instance learned that this thing can easily be configured to serve multiple SSIDs with different security settings from the same radio, and the SSIDs can be attached to separate VLANs. That means I could create one encrypted SSID for my private use, and an open one for passers by.

Anyway, here's a quick configuration example. It creates an 802.11g interface which is WPA-PSK protected and bridged to the 4-port ethernet switch:

interface Dot11Radio0
 no ip address
 !
 encryption mode ciphers tkip
 !
 ssid Koo
    authentication open
    authentication key-management wpa
    guest-mode
    wpa-psk ascii 7 XXXXXSECRETXXXXSTRINGXXXXX
 !
 speed basic-1.0 2.0 5.5 6.0 9.0 11.0 12.0 18.0 24.0 36.0 48.0 54.0
 station-role root
 bridge-group 1
 bridge-group 1 spanning-disabled

IBM SDD driver troubleshooting on Linux

2007-10-26T16:35:00+03:00

The Subsystem Device Driver [SDD] is a pseudo device driver designed to support the multipath configuration environments in the IBM TotalStorage Enterprise Storage Server, the IBM TotalStorage DS family, the IBM SystemStorage SAN Volume Controller. It resides in a host system with the native disk device driver and provides the following functions:

- Enhanced data availability
- Dynamic I/O load-balancing across multiple paths
- Automatic path failover protection
- Concurrent download of licensed internal code
- Path-selection policies for the host system

On AIX it works pretty much out-of-the-box, but on Linux the story is quite different. The driver is closed-source, so all you can do is download the driver kernel module for supported kernels from IBM, as well as some userspace software.

For supported distributions, the driver can be downloaded from:

Subsystem Device Driver for Linux

When you install the package, it copies its files under /opt/IBMsdd (at least the Redhat .rpms do). Also, it adds an init script called "sdd" under /etc/init.d, and adds a line like this in inittab:

srv:345:respawn:/opt/IBMsdd/bin/sddsrv > /dev/null 2>&1

That should take care that if the sddsrv daemon dies, it will be restarted automatically.

Basic configuration

After the disks have been configured in the SAN Volume Controller, they should show up when the busses of the Fibre Channel adapters are re-scanned.

/proc/partitions shows the new disk:

252     0   73400320 vpatha

/etc/vpath.conf holds the names for the device ID's, and /etc/sddsrv.conf some miscellaneous settings.

Under /opt/IBMsdd/bin there are some useful utilities. The datapath command will allow you to query adapters and devices, and set them offline or online. lsvpcg shows which scsi disks map to which vpath disks. cfgvpath allows you to make changes to the configuration.

Using with LVM on RHEL4

To get LVM working correctly with SDD on RHEL4, there are a couple of things that must be taken care of.

Boot-up load order

The first thing is to make sure the sdd driver loads before LVM on boot-up. The SDD User Manual suggests adding a script to start sdd in /etc/rc.sysinit. It must be started after the root filesystem has been remounted but before LVM initializes:

#  Remount  the  root  filesystem  read-write.
update_boot_stage  RCmountfs
state=`awk  /  /  /  &&  ($3  !~  /rootfs/)  {  print  $4  }  /proc/mounts`
[  "$state"  !=  "rw"  -a  "$READONLY"  !=  "yes"  ]  &&
action  $"Remounting  root  filesystem  in  read-write  mode:  "  mount  -n  -o  remount,rw  /

#  Starting  SDD
/etc/init.d/sdd start

#  LVM  initialization
...

Also, they say /etc/init.d/sdd script must be set _not_ to start at boot-up:

[root@server ~]# chkconfig sdd off

The above configuration has a problem, though. The SDD software is installed under /opt, but if your /opt is on an LVM partition, there is no way to load the SDD drivers from there before LVM has been started and /opt mounted. In fact, if the line sits there, it will not work if /opt resides on its own partition of any type.

One solution to that problem is to move the SDD drivers to the root partition. For me it seems to just work if I enable /etc/init.d/sdd at bootup (no changes needed in rc.sysinit):

[root@server ~]# chkconfig sdd on

This may be due to the fact that my root partition is also in LVM, so that it has to be initialized in initrd early, and perhaps LVM scans for devices again later on.

Volume initialization and detection problems

A couple of changes must be made to the LVM configuration file /etc/lvm/lvm.conf. If you don't do that, you will run into an error message like this while trying to create a physical volume on a vpath device:

[root@server ~]# pvcreate /dev/vpatha1
  Device /dev/vpatha1 not found (or ignored by filtering).

Accepting vpath devices

In /etc/lvm/lvm.conf, you will see that this line is commented out:

# types = [ "fd", 16 ]

Remove the comment character, and change the line to look like this:

types = [ "vpath", 16 ]

That will add vpath to LVM's list of accepted device types.

Rejecting the underlying devices

The lvm.conf has a filter configuration option which is used to select the devices that are allowed to be used as LVM physical volumes. The default is usually to allow every device. That is a problem with vpath devices, however, because they will show up both as /dev/vpathX, and as regular scsi disks. So, if you have configured one disk per volume controller through two FC switches, you will see a total four /dev/sdX disks per one vpath disk. And all these five devices will show the same data (although only the vpath device is redundant, and should therefore be used). The LVM default is to scan all devices, so it will print error messages of duplicate physical volumes if you don't change the filtering rule. The messages look like this:

Found duplicate PV 1XlJrZHnI49tTtHVvwe7cXZ0cATNFTxw: using /dev/sdak not /dev/vpatha

This is particularly bad, as it means LVM has chosen to use sdak instead of the redundant vpatha path.

To prevent this, change the filter line in lvm.conf to look like this:

filter  =  [  "a/vpath[a-z]*/",  "r/.*/"  ]

That line means LVM accepts only devices named vpath[a-z]*. This way LVM will choose the correct device. However, if you have physical volumes on internal SCSI disks, that line will also reject them. Add an accept rule on those:

filter  =  [  "a/vpath[a-z]*/", "a/sda2/", "r/.*/"  ]

After these changes, you should be able to create a physical volume on vpath devices, and no error messages should be displayed when handing volumes. And everything should look the same the next time you boot.

Bootstrapping an Ubuntu guest for Xen

2007-10-24T12:53:00+03:00

First, some empty disk space is needed. Let's create a logical volume for our new virtual machine:

root@xenserver1:~# lvcreate -n testlv -L 10G vg0
  Logical volume "testlv" created

Create a filesystem on the new logical volume:

root@xenserver1:~# mke2fs -j /dev/vg0/testlv
mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1310720 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 25 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Create a mount point:

root@xenserver1:~# mkdir /mnt/test

Mount the new filesystem

root@xenserver1:~# mount /dev/vg0/testlv /mnt/test

Bootstrap the new system. For this, we will use a utility called "debootstrap", which can bootstrap a fresh Ubuntu installation easily:

root@xenserver1:~# apt-get install debootstrap
root@xenserver1:~# debootstrap feisty /mnt/test

After the system is bootstrapped, kernel modules must be copied inside the guest system for it to be able to load them during startup:modules:

root@xenserver1:~# cp -a /lib/modules/2.6.19-4-server /mnt/test/lib/modules/

Guest system's /etc/fstab must be created:

root@xenserver1:~# nano -w /mnt/test/etc/fstab

proc            /proc           proc    defaults        0       0
/dev/sda1       /               ext3    defaults,errors=remount-ro 0       1

Give the guest a hostname:

root@xenserver1:~# echo "test" > /mnt/test/etc/hostname

Umount the filesystem so that we can boot from it:

root@xenserver1:~# umount /mnt/test/

Create a Xen configuration file:

root@xenserver1:/etc/xen/auto# nano -w /etc/xen/auto/test

kernel = "/boot/vmlinuz-2.6.19-4-server" # Your kernel image
ramdisk = "/boot/initrd.img-2.6.19-4-server" # The initial ramdisk
memory = 128 # Number of megabytes allocated
name = "test" # Virtual machine name
vcpus = 4 # Number of virtual CPUs the guest sees
vif = [ 'mac=aa:00:00:00:13:13, bridge=xenbr0' ] # A network interface
disk = [ 'phy:mapper/vg0-testlv,sda1,w' ] # The root disk
root = "/dev/sda1 ro" # Root partition the kernel mounts first

Boot the machine:

root@xenserver1:~# xm create -c /etc/xen/auto/test

Log in:

Ubuntu 7.04 test tty1

test login: root

The root password is initially empty, which is not very secure. Change root password:

root@test:~# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

Configure network:

root@test:~# vi /etc/network/interfaces

auto lo eth0
iface lo inet loopback
iface eth0 inet dhcp

root@test:~# /etc/init.d/networking restart

Install some useful packages:

root@test:~# apt-get update
root@test:~# apt-get install language-pack-fi
root@test:~# apt-get install nano cron bash man
root@test:~# apt-get install openssh-server

That's it! You now have a working virtual machine ready for use.

How to get VMware Server working with an unsupported kernel and the vmware-any-any patch

2007-10-17T20:23:00+03:00

VMware Server needs exactly two kernel modules running on the host system (there are separate modules for guest systems). These are the vmmon and the vmnet modules. Unfortunately, the vmmon and vmnet packages included in the VMware server distribution package don't compile with the newest kernels. When I upgraded my laptop to Gutsy Gibbon a few weeks ago, I forgot to check if VMware server supports the new kernel. And, of course, it doesn't yet. But luckily I found the vmware-any-any package, a patched version of the host kernel modules that works with newer kernel versions.

You can download the package from here:

http://knihovny.cvut.cz/ftp/pub/vmware/?M=D

Download the tarball and unpack it:

mkortela@T60:~/Desktop$ tar zxvf vmware-any-any-update114.tar.gz
vmware-any-any-update114/
vmware-any-any-update114/services.sh
vmware-any-any-update114/runme.pl
vmware-any-any-update114/update.c
vmware-any-any-update114/vmnet.tar
vmware-any-any-update114/update
vmware-any-any-update114/vmblock.tar
vmware-any-any-update114/vmmon.tar

You also need the source code of your running kernel. On Ubuntu, you can get it by running:

mkortela@T60:~/Desktop$ sudo apt-get install linux-source

There's a "runme.pl" script included which should be able to compile the modules and patch your system, but I couldn't get it working. Instead, I unpacked the the vmmon.tar and vmnet.tar and compiled them myself:

mkortela@T60:~/Desktop/vmware-any-any-update114$ tar xf vmmon.tar
mkortela@T60:~/Desktop/vmware-any-any-update114$ tar xf vmnet.tar
mkortela@T60:~/Desktop/vmware-any-any-update114$ cd vmmon-only
mkortela@T60:~/Desktop/vmware-any-any-update114/vmmon-only$ make
.
.
.
mkortela@T60:~/Desktop/vmware-any-any-update114$ cd ../vmnet-only
mkortela@T60:~/Desktop/vmware-any-any-update114/vmnet-only$ make
.
.
.

The result is two files which can be loaded into the kernel as modules: vmmon.ko and vmnet.ko. You can simply insert those into the kernel with the insmod command:

mkortela@T60:~/Desktop/vmware-any-any-update114$ sudo insmod vmnet-only/vmnet.ko
mkortela@T60:~/Desktop/vmware-any-any-update114$ sudo insmod vmmon-only/vmmon.ko

After that, VMware server starts up nicely!

But one thing is still left. To be able to load the modules at startup properly, they must be copied to the kernel module tree under /lib. First I created a directory for the modules, and then copied them there. After that operation, the command depmod must be run in order to create an up-to-date list of available modules:

mkortela@T60:~/Desktop/vmware-any-any-update114$ sudo mkdir /lib/modules/2.6.22-14-generic/vmware-server
mkortela@T60:~/Desktop/vmware-any-any-update114$ sudo cp vmmon-only/vmmon.ko /lib/modules/2.6.22-14-generic/vmware-server/
mkortela@T60:~/Desktop/vmware-any-any-update114$ sudo cp vmnet-only/vmnet.ko /lib/modules/2.6.22-14-generic/vmware-server/
mkortela@T60:/lib/modules/2.6.22-12-generic/vmware-server$ sudo depmod

That's it! Now you should be able to run VMware server, and all the modules should be loaded at runtime. Please note that if you are doing a fresh install of VMware server, you must now run the vmware-config-network.pl script to configure the network.

You can make sure that the modules load properly by unloading them with rmmod and then loading them again with modprobe:

mkortela@T60:/lib/modules/2.6.22-12-generic/vmware-server$ sudo rmmod vmmon
mkortela@T60:/lib/modules/2.6.22-12-generic/vmware-server$ sudo rmmod vmnet
mkortela@T60:/lib/modules/2.6.22-12-generic/vmware-server$ sudo modprobe vmmon
mkortela@T60:/lib/modules/2.6.22-12-generic/vmware-server$ sudo modprobe vmnet

Vmmon should print a status message like this in your dmesg:

[  999.429124] /dev/vmmon[8602]: VMCI: Driver initialized.
[  999.432385] /dev/vmmon[8602]: Module vmmon: registered with major=10 minor=165
[  999.432756] /dev/vmmon[8602]: Module vmmon: initialized

Updated version of the flash64.sh script for Ubuntu 7.10 Gutsy Gibbon

2007-10-02T06:28:00+03:00

A few months ago I released a script to install a 32-bit Adobe Flash plugin to a 64-bit Firefox, but it doesn't seem to work in Gutsy beta. Here's an updated one. I upgraded my laptop to Gutsy and used this one to get Flash working.

Edit: I did a complete reinstall later and learned that Gutsy knows how to set this up by itself. So there's no need for this script anymore. And that is a very positive thing!

Just cut and paste it to "flash64.sh", add execute permissions, and run it with sudo.

#!/bin/sh

DEPS="ia32-libs ia32-libs-gtk util-linux lib32asound2 alien"
GWENOLE=http://gwenole.beauchesne.info/projects/nspluginwrapper/files
PLUGIN=nspluginwrapper-0.9.91.4-1.x86_64.rpm
VIEWER=nspluginwrapper-i386-0.9.91.4-1.x86_64.rpm
MACROMEDIA=http://fpdownload.macromedia.com/get/flashplayer/current
FLASH=install_flash_player_9_linux.tar.gz

usage() {
  cat << EOF
A script to install 32-bit Adobe Flash plugin in a 64-bit Firefox.
Usage: $0 [-k]
       -k   keep temporary files
EOF
}

while getopts "kh?" OPT
do
  case $OPT in
    "k") KEEPTMP=1;;
    "h"|"?"|"help") usage; exit;;
    *) echo Unrecognized option "$OPT"; usage; exit;;
  esac
done

if [ "`id -u`" != "0" ]
then
  echo You must be root in order to run this script.
  echo Try: "sudo $0" instead.
  echo For options and help, use "$0 -h".
  exit 1
fi

TMP=`mktemp -d`
cd $TMP
echo Temporary directory $TMP created

echo Installing dependencies: $DEPS
apt-get -qq install $DEPS || exit 1

echo Getting nspluginwrapper plugin: $PLUGIN
wget -qc $GWENOLE/$PLUGIN

echo Getting nspluginwrapper viewer: $VIEWER
wget -qc $GWENOLE/$VIEWER

echo Installing plugin
alien -i $PLUGIN

echo Installing viewer
alien -i $VIEWER

echo Getting Flash plugin: $PLUGIN
wget -qc $MACROMEDIA/$FLASH

echo Installing Flash plugin to /usr/lib32/mozilla/plugins
mkdir -p /usr/lib32/mozilla/plugins
tar -Ozxf $FLASH install_flash_player_9_linux/flashplayer.xpt > /usr/lib32/mozilla/plugins/flashplayer.xpt
tar -Ozxf $FLASH install_flash_player_9_linux/libflashplayer.so > /usr/lib32/mozilla/plugins/libflashplayer.so

echo Creating a wrapper using nspluginwrapper
nspluginwrapper -i /usr/lib32/mozilla/plugins/libflashplayer.so

if [ "$KEEPTMP" = "1" ]
then
  echo Temporary files were left in: $TMP
else
  echo Removing temporary files
  rm -rf $TMP
fi

echo
echo The Flash plugin and wrapper have been installed.
echo You must restart Firefox.

Getting a Certificate for your Web Server

2007-10-01T21:50:00+03:00

To communicate securely using SSL (also known as TLS or Transport Layer Security), web servers need a key pair of public and private keys. This key pair can be generated and signed by yourself, but to prevent the web browser from asking "stupid" questions about the validity of a certificate, you must pay a company, called a Certificate authority, such as Verisign, Geotrust, or Thawte, for the added simplicity. That company will then verify that you are who you are (in theory, at least) and then sign your public key with their certificate. Their certificate is already bundled with most browsers and thus trusted by default, which also makes your newly signed certificate trusted as well.

In short, the steps needed to get and install a key that is signed by a certification authority:

Create an a public/private key pair
Create a Certificate Signing Request
Submit your Certificate Signing Request to your Certificate Authority
The CA will check your identity with a method they have chosen
The CA will give back your certificate signed with theirs
Copy the key file and the certificate file to your web server
Configure your web server to use the new certificate

A Real World Example

Now let's go through the steps given above with a real example. I have OpenSSL, and specifically the openssl command line tool, installed on my machine, which is nice because it is one of the best tools available for handling SSL/TLS keys and certificates. If you are running a typical Linux installation, you will most likely have it installed already.

1. Create a public/private key pair

The following command will create an RSA private key to the file specified, in this case koo.fi.key.

mkortela@T60:~$ openssl genrsa 1024 > koo.fi.key
Generating RSA private key, 1024 bit long modulus
.....................................++++++
...++++++
e is 65537 (0x10001)

Remember that this file contains private information which must be kept very secret. In fact, after completing step 2, the only one who should have even read access to the file is the web server daemon.

2. Create a Certificate Signing Request

Next we will use the openssl tool again to create the signing request, which will contain some information about us, as well as the public key of the key pair. This time OpenSSL will ask us some questions about the user of the certificate. This info should be as accurate as possible, but in most cases, it doesn't really matter what you write there. Usually the only field that has any meaning is the "Common Name" field, which should be the same as the domain name of your web server (www.yourdomain.com). Here we will use the common name koo.fi. The CSR will be output to the file koo.fi.csr.

mkortela@T60:~$ openssl req -new -key koo.fi.key -out koo.fi.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:FI
State or Province Name (full name) [Some-State]:Uusimaa
Locality Name (eg, city) []:Helsinki
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Techelp Oy
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:koo.fi
Email Address []:mikko.kortelainen@techelp.fi

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

3. Submit your Certificate Signing Request to your Certificate Authority

Now that we have one file for our private key and another one for the certificate signing request, it is time to pay a commercial CA a number of euros for signing our certificate.

I have been using RapidSSL.com, because they can provide me quite well accepted certificate quite cheaply and instantly. In fact, you don't even have to speak to any service personnel, because the authentication telephone call is automated.

So click on the "BUY" button and answer all questions. When it is time to give the Certificate Signing Request, cut and paste the contents of your CSR file (_NOT_ the key file!).

4. The CA will check your identity with a method they have chosen

RapidSSL will give you and automated phone call. Follow instructions.

5. The CA will give back your certificate signed with theirs

In a couple of minutes you will get an e-mail containing the the certificate. Look for something like this:

-----BEGIN CERTIFICATE-----
.
.
... a lot of gibberish ...
.
.
-----END CERTIFICATE-----

Cut and paste that to a new file, mine was called koo.fi.crt. This file now contains the signed certificate.

6. Copy the key and the certificate to your web server

The last thing to do is to copy the key file and the signed certificate file (the one you got from your CA) to the web server in a location the web server will find them. I copied my certificate to:

/etc/ssl/koo.fi.crt

And my private key to:

/etc/ssl/private/koo.fi.key

Then I changed the permissions like this:

chown root:root /etc/ssl/koo.fi.crt
chown root:root /etc/ssl/private/koo.fi.key
chmod 644 /etc/ssl/koo.fi.crt
chmod 400 /etc/ssl/private/koo.fi.key

7. Configure your web server to use the new certificate

The last step is, of course, to configure your web server to use the certificate. For Apache 2 you could add something like this to your configuration file:

<Virtualhost *:443>
    ServerName koo.fi
    ServerAlias koo.fi
    DocumentRoot /path/to/documentroot
    ServerAdmin mikko@koo.fi

    SSLEngine on
    SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
    SSLCertificateFile /etc/ssl/koo.fi.crt
    SSLCertificateKeyFile /etc/ssl/koo.fi.key
</VirtualHost>

What next?

The next step is, of course, running your own Certification Authority. That is somewhat of an advanced topic, but perhaps you will see an article discussing it on this site later on.

Before you go, though, you really should try the following commands on your key, signing request and certificate files:

openssl rsa -noout -text -in koo.fi.key
openssl req -noout -text -in koo.fi.csr
openssl x509 -noout -text -in koo.fi.crt

Those will give you a readable version of the contents. That can be really handy if you're in a hurry and forgot which file was which.

Installing OpenSSH server on a Windows box

2007-08-21T13:35:00+03:00

An SSH server can be handy on a Windows machine, too. Cygwin comes with OpenSSH, and provides a lot of useful tools which you can use over the SSH connection. Here's how to install Cygwin and OpenSSH server on a Windows machine.

Install Cygwin. Make sure you also install OpenSSH under the Net category. You can download the installer from here:

http://www.cygwin.com/setup.exe

To successfully run OpenSSH on a Windows 2003 Server, you must create a new user account for it. This is because the SYSTEM account, which is the default when installing OpenSSH as a service, does not have the "Create a token object" right, which is needed for public key authentication. Luckily, the ssh-host-config command will create a user for you, if you wish. Just remember to select "yes" in the following prompts.

In fact, two user accounts are created in the script below. The other one is for privilege separation, which will make your installation a bit more secure. There's no reason not to enable it.

If you are installing OpenSSH on an Active Directory domain controller, the user accounts will be created in Active Directory. This is because a domain controller does not have a separate local user database, but the local user database is a copy of AD itself. Remember this if you are going to install OpenSSH on more than one DC. On the second installation, the user accounts are already created!

administrator@server ~
$ ssh-host-config
Generating /etc/ssh_host_key
Generating /etc/ssh_host_rsa_key
Generating /etc/ssh_host_dsa_key
Generating /etc/ssh_config file
Privilege separation is set to yes by default since OpenSSH 3.3.
However, this requires a non-privileged account called 'sshd'.
For more info on privilege separation read /usr/share/doc/openssh/README.privsep.

Should privilege separation be used? (yes/no) yes
Warning: The following function requires administrator privileges!
Should this script create a local user 'sshd' on this machine? (yes/no) yes
Generating /etc/sshd_config file

Warning: The following functions require administrator privileges!

Do you want to install sshd as service?
(Say "no" if it's already installed as service) (yes/no) yes

You appear to be running Windows 2003 Server or later.  On 2003 and
later systems, it's not possible to use the LocalSystem account
if sshd should allow passwordless logon (e. g. public key authentication).
If you want to enable that functionality, it's required to create a new
account 'sshd_server' with special privileges, which is then used to run
the sshd service under.

Should this script create a new local account 'sshd_server' which has
the required privileges? (yes/no) yes

Please enter a password for new user 'sshd_server'.  Please be sure that
this password matches the password rules given on your system.
Entering no password will exit the configuration.  PASSWORD=SECRET

User 'sshd_server' has been created with password 'SECRET'.
If you change the password, please keep in mind to change the password
for the sshd service, too.

Also keep in mind that the user sshd_server needs read permissions on all
users' .ssh/authorized_keys file to allow public key authentication for
these users!.  (Re-)running ssh-user-config for each user will set the
required permissions correctly.

Which value should the environment variable CYGWIN have when
sshd starts? It's recommended to set at least "ntsec" to be
able to change user context without password.
Default is "ntsec".  CYGWIN=binmode ntsec tty

The service has been installed under sshd_server account.
To start the service, call `net start sshd' or `cygrunsrv -S sshd'.

Host configuration finished. Have fun!

You should now have a new service, called "CYGWIN sshd", installed. It doesn't start automatically, so you must start it either from the Windows Services MMC Console or from the command line with the command "net start sshd".

Creating a RAID array out of cheap USB disks

2007-08-05T15:33:00+03:00

A RAID array is a bunch of disks used together in co-operation to create a redundant storage facility. Hard drives are mechanical devices with moving parts, which makes them prone to failure. You can imagine what it means to spin the hard drive platters at 7200 rpm, for years without a pause in a typical server setup. Eventually, all disks die. A disk may run for a decade without a hinge, but another disk of the same type may die after a year. The problem is, there is reliable way to predict when a particular disk dies.

To fight this problem, a number of schemes have been devised. They collectively go by the name of RAID, or Redundant Array of Inexpensive disks. The common denominator for them is that they all use one or more extra disks to hold a second copy of the data, so that the failure of one disk will not interrupt service. An exception to this is RAID level 0, also known as striping, which really is no RAID at all, because it does not provide any kind of redundancy. As opposed to using the disks separately, striping will in fact increase the probability of a fatal disk crash. If one drive fails, the data on all drives will be lost. Striping is therefore useful only for temporary data storage.

A failed disk should be replaced as soon as possible, because in a typical situation the array can not sustain another disk failure before a new disk has been added and the array rebuilt (with the notable exception of RAID 6). The good thing is, this all can be done without having to put the data offline.

In this article I am going to show you how to create a RAID 5 array out of ordinary USB hard disk drives. Later on, I will show you how to extend the array with another disk to add some more space. The good thing about these drives is that they are cheap, and hot-pluggable. The result will obviously not be as fast as an array created out of SATA drives, but it serves well for storing your music, videos, backups etc. Running the command "hdparm -t" for my RAID array made out of SATA drives will give me a reading of about 68 MB/sec, as opposed to the USB array, which gets on average about 42 MB/sec. For many applications this is quite enough.

By the way, the commands in this article apply to other kinds of drives as well. In fact, the USB hard disks show up as regular SCSI or SATA disks. You can create a RAID array from all kinds of block devices, and even mix them. You could, for example, create a RAID 1 out of two SATA disks, and then add one USB disk to the array as a spare.

Setting up the array

That's enough theory, now it is time to get our hands dirty. In the beginning, my /proc/partitions had lines like this:

major minor  #blocks  name
   8    32  488386584 sdc
   8    48  488386584 sdd
   8    64  488386584 sde
   8    80  488386584 sdf

Those were the four USB hard drives I wanted to use for my array.

Before creating the array, the individual disks must be prepared. First, you must create a single primary partition on each disk, and set the partition type to 0xFD - "Linux RAID autodetect". This way both you and the kernel will easily see that the partition belongs to a RAID array.

Next, the command to create an array of three disks looks like this:

mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdc1 /dev/sdd1 /dev/sde1

You should now have a device called /dev/md0. That is the disk you are going to have to use from now on. To see the details of your newly created array, run:

mdadm --detail /dev/md0

The command will show you some basic information about your array, as well as status information for each of the disks. A RAID 5 array will continue to work even if at most one drive has failed. When a new, working drive is inserted into the array, it will take some time to copy the relevant data to the new drive. This will be shown as the "rebuild status" in the detailed listing. For example my 500GB USB hard drives take several hours to rebuild.

You can get some interesting information also by looking at the file /proc/mdstat. That file comes straight from the kernel, and contains some very basic info. For a rebuilding array, this file contains an estimate of the time left. Your new md device should also show up in /proc/partitions now.

To take advantage of our redundant array, a filesystem must be created on it. You can no more use the /dev/sdX devices directly, as that would corrupt the array. You must use the device /dev/md0 instead.

Logical Volume Management

If you want, you can create a filesystem directly on your md device, but I would recommend using LVM2 instead. LVM stands for Logical Volume Manager, and it is quite a powerful tool for managing disks. To cut a long story short, it can be used to create, resize and move partitions very flexibly (on-line in almost every case), in addition to a number of other things.

Make sure you have the LVM2 userland tools installed. For Ubuntu, "apt-get install lvm2".

First thing to do is create a physical volume out of our new md device:

pvcreate /dev/md0

Step number two is to create a volume group which will act as a container for our logical volumes:

vgcreate myvg /dev/md0

myvg is just a name for our new volume group. You can use whatever name you want. It is used for distinguishing between volume groups, as there can be more than one in a given system.

A volume group is just an abstraction used to group one or more physical volumes into a single space which can be divided into logical volumes. A logical volume looks just like a disk partition to the kernel, and so a filesystem can be created on it.

Let's take a look first at our volume group. Running the command vgs will give you a quick list of your volume groups:

VG      #PV #LV #SN Attr   VSize   VFree
myvg      1   2   0 wz--n- 931,52G 931,52G

As you see, you have now about 930 GB free space on your volume group. Next, let's create a 100 GB logical volume:

lvcreate --size 100G --name mylv myvg

The command lvs will give a list of volumes in the system:

LV       VG      Attr   LSize   Origin Snap%  Move Log Copy%
mylv     myvg    -wi-ao 100,00G

The logical volume will now show up as the device /dev/myvg/mylv. To create a file system on it:

mke2fs -j /dev/myvg/mylv

You can now mount it anywhere you want. Let's create a mount point in /myfs and mount the new logical volume there:

mkdir /myfs
mount /dev/myvg/mylv /myfs

Add a line to /etc/fstab to mount it automatically each boot.

By the way, in addition to the commands pvs, vgs and lvs, you can get more detailed information about the physical and logical volumes, and volume groups, with the commands pvdisplay, vgdisplay and lvdisplay.

Extending your array

The newest version of RAID support in the kernel includes support for extending a RAID 5 array. What that means is if you are running out of space, you can just plug in a new USB disk, extend the array, and then extend your filesystem. All this can be done with the filesystem online, and without ever rebooting your machine. Here's how.

As you probably noticed, I intentionally left the fourth disk out of the array when first creating it. We will now add the fourth disk there.

The growing process consists of two commands. The first command will add the new disk to the array, making it a spare drive, and the second one will grow the array to the new size:

mdadm /dev/md0 --add /dev/sdf1
mdadm --grow /dev/md0 --raid-disks 4

You can now see with "mdadm -D /dev/md0" or "cat /proc/partitions" that the array is rebuilding. The reshape process will likely take several hours, and the new space will not be available before that:

md0 : active raid5 sdf1[0] sdc1[3] sdd1[2] sde1[1]
      1465159488 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [==================>..]  check = 94.3% (460724960/488386496) finish=39.1min speed=11763K/sec

After the reshape is complete, you can add the new free space to the physical volume by running:

pvresize /dev/md0

Pvresize will by default resize the physical volume to fill all available space on the disk. And this is just what we want. vgs will now show that we have more free disk space in myvg:

VG      #PV #LV #SN Attr   VSize   VFree
myvg      1   2   0 wz--n- 1,36T   1,26T

To add a terabyte to our logical volume:

lvextend --size +1T /dev/myvg/mylv

The last step is to increase the filesystem. A tool called ext2online can be used to extend the filesystem without umounting it. Install it with "apt-get install ext2resize". Run the command like this:

ext2online /dev/myvg/mylv

And that's it. You have now added more free space to your filesystem without disrupting operation.

Detecting problems and managing failed disks

The mdadm tool includes a monitor mode which can be used to monitor the array, detect problems and let you know about them. On an Ubuntu system, a configuration file exists in /etc/mdadm/mdadm.conf. You can put your e-mail address on the MAILADDR line to receive e-mail when mdadm --monitor detects a problem. Just make sure your system can send e-mail properly.

If a disk dies and you have set up a spare disk, the kernel will automatically rebuild the array using the spare. Otherwise the array will remain usable in a degraded state, but will not sustain another disk failure before you add more disks to it.

You can remove the failed disk from the array with:

mdadm /dev/md0 --remove /dev/sdX

To add another disk to the array, use:

mdadm /dev/md0 --add /dev/sdX

If the array is missing a disk, the disk will automatically be used for rebuilding it. If the array is complete, the disk will be added as a spare.

If you intentionally want to replace a drive with another one in a working array, run:

mdadm /dev/md0 --add /dev/sdX --fail /dev/sdY --remove /dev/sdY

This will add disk /dev/sdX to the array and remove /dev/sdY from it. Disk sdY must be marked as failed to be able to remove it.

Installing the 32-bit Adobe Flash plugin on a 64-bit Firefox in Feisty

2007-07-29T21:40:00+03:00

Here's a little script I made to make installing the 32-bit Flash plugin on a 64-bit Firefox easy. It was adapted from a script I found at the Ubuntu forums (here), but since that did not work right out of the box for me, I created one that does. The script uses the nspluginwrapper tool by Gwenol

Starting and stopping IBM Websphere MQ on a Redhat Enterprise Linux box

2007-07-25T18:30:00+03:00

Here's a little script I made for starting and stopping Websphere MQ on a RHEL 4. It is useful for a simple configuration with one Queue Manager and listener. Just put your queue manager name in the QMGR variable and the listener port to the PORT variable, and save the script to your /etc/init.d.

Please understand that this script does not check if the processes actually stay up or die as they should. But it is useful for automating the tasks during a reboot.

If you are looking for a more powerful administration tool for MQ, check out Dr. Johannes B

Site opened

2007-07-24T22:03:00+03:00

Welcome to Mikko's home page. There's nothing much here quite yet, but I'm working on it.