shad.cc

CirruxCache: Store API documentation

shad — Sat, 07 Aug 2010 19:29:00 +0200

I have just written some documentation about the Store WebService of CirruxCache. This little page will help you to upload big files on CirruxCache in the same way than the current admin panel does.

http://code.google.com/p/cirruxcach...

Just to remember, the Store WebService has been developed to bypass the 1MB limitation of appengine. It is not possible to dynamically cache objects with a size greater than 1MB, in this case, you need to store explicitly such an object using this WebService. It uses the Blobstore service, so file size cannot exceed 2GB.

CirruxCache: Advanced configuration sample

shad — Wed, 21 Jul 2010 22:46:00 +0200

That's it! This whole blog is cached and directly delivered by CirruxCache (only static files were cached before). My origin server is a tiny eeebox connected through my personal ISP.

So this configuration is a good challenge to offload my tiny web server as much as possible.

I think this is a good opportunity to show an example of a configuration which is a little bit more evolved.

The point is, I cannot set the same cache TTL for the whole website, and actually, I want to cache several websites...

# URL mapping
urls = {}
 
base = (
                '/_admin/(.*)', 'Admin',
                '/_store/(.*)', 'Store',
                '/_cron/(.*)', 'Cron'
                )
 
urls['default'] = base + (
                '(/debug/.*)', 'Debug',
                '/(.*)', 'Root'
                )
 
urls['www.shad.cc'] = base + (
                '(/themes/.*)', 'Blog_Static',
                '(/plugins/.*)', 'Blog_Static',
                '(/admin/.*)', 'Blog_Forward',
                '(/.*)', 'Blog_Page'
                )
 
urls['www.zaphod.eu'] = base + (
                '(/pub/.*)', 'Zaphod_Redirect',
                '(/.*)', 'Zaphod'
                )
 
# still supporting the old config
 
urls['cdn.shad.cc'] = base + (
                '/blog(/.*)', 'Blog_Static',
                '/(.*)', 'Root'
                )
 
urls['cdn.zaphod.eu'] = base + (
                '(/admin/.*)', 'Zaphod',
                '/(.*)', 'Root'
                )
 
# POP definition
# You can define and configure your Point Of Presence
 
class Blog_Static(cache.Service):
        origin = 'http://orig.shad.cc'
        forceTTL = 2592000 # 1 month
        ignoreQueryString = True
        forwardPost = False
        allowFlushFrom = ['x.x.x.x']
 
class Blog_Page(cache.Service):
        origin = 'http://orig.shad.cc'
        forceTTL = 3600 # 1 hour
        ignoreQueryString = True
        forwardPost = True
        allowFlushFrom = ['x.x.x.x']
 
class Blog_Forward(forward.Service):
        origin = 'http://orig.shad.cc'
 
class Zaphod(cache.Service):
        origin = 'http://orig.zaphod.eu'
        forceTTL = 2592000 # 1 month
        ignoreQueryString = True
        forwardPost = False
        allowFlushFrom = ['x.x.x.x']
 
class Zaphod_Redirect(redirect.Service):
        origin = 'http://zaphod.eu'
 
# !POP

I think this configuration is enough readable to avoid any explanation. However, do not hesitate to leave any comments.

Finally, I created a google groups to centralize all help requests. So if you need help, go to http://groups.google.com/group/cirr... or send an email to cirruxcache 'at' googlegroups 'dot' com.

New release: CirruxCache 0.3.1

shad — Sun, 18 Jul 2010 15:21:00 +0200

I am really glad to announce a new major release of CirruxCache.

This new release includes the following changes:

A storage webservice: store big files (<= 2GB) on the Blobstore in order to deliver them through CirruxCache. This feature is useful to bypass the 1MB limit on appengine.
An admin panel that enables users to flush objects, manage big files and see some statistics about the resources used.
Bugfixes

It is really important to note there are few limitations on the panel admin:

There is no error reporting on the flush panel (it only displays the number of objects trying to be flushed).
Storage manager displays a "500 Internal Error" when uploading. It only happens when you don't have a billing account (the Blobstore is only available on billing accounts, refer to appengine).

These two limitations will be improved in the next release, and there will be more informations in the statistics panel.

The Storage WebService will be documented really soon, but you can access the admin panel through "http://your.cirruxcache.app/_admin/"

I make the most of this opportunity to announce some changes on the project website:

Roadmap page
Users list: this page has just been started, so do not hesitate to notify me to add you.

I hope you will enjoy this new release.

Minor release: CirruxCache 0.2.2

shad — Wed, 19 May 2010 17:45:00 +0200

CirruxCache 0.2.2 has just been released. It contains some bugfixes (thanks to Devattas to have reported errors on Datastore latency). Webpy has been updated to the last version.

I have also updated the documentation, especially I brought more details on Point of Presence configuration and usage of cron tasks for garbage collection.

Finally, some of the users reported me that there is a real problem with the cached object size limit (currently 1MB). I am working on the solution, I will take advantage of the new Blobstore service on AppEngine to store objects. Maybe I will keep the Datastore only for meta-data. This solution will raise the cache object limit to 50MB.

Stay tuned

CirruxCache 0.2.1 is released

shad — Thu, 11 Mar 2010 17:20:00 +0100

I have just released a new version (0.2.1) of CirruxCache. To remember:

CirruxCache provides a software solution to dynamically cache HTTP objects on Google Appengine (using the Datastore and the Memcache services).

This new version includes an interesting set of features:

allow object flushing from restricted IP
configure a PoP (Point of Presence) according to a virtual host
several behaviors (cache, redirect, forward)

In more details, the last feature is the ability to configure a point of presence to differ from a classical caching mechanism. For example, I may want to configure "/admin/*" on my website to be redirected on the origin without caching.

Of course, this release includes several bugfixes, especially a fix on the "Expires" HTTP header which improves the caching performances.

Do not hesitate to test this new version and to comment any bugs or any suggestions.

Adding virtual host support to webpy

shad — Fri, 19 Feb 2010 18:11:00 +0100

Webpy is a tiny web framework. I use it a lot for my web-services applications.

In general, I let my web server (lighttpd) to handle virtual hosting. But as you may know, I am working on a CDN solution on top of Google App Engine, named CirruxCache. In that case, while I have absolutely no control on the server configuration, I need to handle virtual hosting from the code.

Webpy maps urls by iterating through a tuple. So my solution is quite simple: wrapping the tuple to override the __iter__ function according to an environment variable (HTTP_HOST).

Let's take this basic webpy example, without vhosting:

import web
 
urls = ('/(.*)', 'hello')
 
class hello(object):
    def GET(self, name):
        if not name:
            name = 'World'
        return 'Hello, %s' % name
 
if __name__ == "__main__":
   app = web.application(urls, globals()) 
   app.run()

Let's add the VhostMapper class:

import web
 
urls = {
      'default' : ('/(.*)', 'hello'),
      'my-vhost.domain.tld' : ('/(.*)', 'helloVhost')
      }
 
class hello(object):
    def GET(self, name):
        if not name:
            name = 'World'
        return 'Hello, %s !' % name
 
class helloVhost(object):
    def GET(self, name):
        return 'Hello %s' % web.ctx.environ['HTTP_HOST']
 
class VhostMapper(object):
    def __iter__(self):
        url = urls['default']
        if 'HTTP_HOST' in web.ctx.environ:
            vhost = web.ctx.environ['HTTP_HOST']
            if vhost in urls:
                url = urls[vhost]
        return iter(url)
 
if __name__ == "__main__":
   app = web.application(VhostMapper(), globals()) 
   app.run()

Finally, you can use curl or wget to test your vhosts:

$> curl -H "Host: my-vhost.domain.tld" http://localhost:8080/

It is not so early to announce that the next version of CirruxCache will handle virtual hosting

I am sure this simple hack can be easily reproduced to use virtual hosting in some other Rest frameworks.

Exploiter le SIP de sa Freebox

shad — Thu, 04 Feb 2010 12:53:00 +0100

J'ai commencé à jouer récemment avec le compte SIP de ma Freebox.

Le compte SIP se paramètre via le panel de gestion de Free, dans l'onglet Téléphonie. Deux options sont possibles:

Rediriger les appels vers la Freebox
Rediriger les appels vers le serveur SIP

Dans le deuxieme cas, il vous faudra un téléphone IP si voulez conserver un téléphone physique connecté à la Freebox. Pour ma part, ayant un téléphone analogique standard, je redirige les appels vers la Freebox. J'utilise donc le serveur SIP uniquement pour passer des appels, et non en recevoir.

Il existe de nombreux clients SIP. J'utilise Linphone sur Linux que je trouve léger et stable.

Un point interessant, le serveur SIP est accessible de l'exterieur, autrement dit, vous pouvez appeler en illimités sur des fixes en France et dans certains pays depuis n'importe où en utilisant votre compte SIP Freephonie.

De plus, si vous êtes l'heureux possesseur d'un iPhone (ou tout autre téléphone sur Symbian, Windows Mobile ou Android), il vous suffit d'installer un client SIP (j'utilise iSipSimple sur iPhone) et de solliciter votre forfait data illimité (ou pas) pour passer vos appels vers les fixes tout en utilisant votre généreux forfait Free.

Attention aux appels vers les mobiles qui ne seraient pas avantageux.

How to auto-update Chromium on Mac OS X, Linux and other

shad — Wed, 06 Jan 2010 12:23:00 +0100

I want to try to use Chromium as my first web browser. But for the moment, Chromium suffers from a lack of package maintaining though it targets developers.

On my Gentoo Linux, I let portage to handle my Chromium version. But I also use Mac OS X at work, so I have written a little shell script to auto update my Chromium.app.

Put the content of this scripts wherever you want and make it executable ($> chmod 755 chromium_update.sh).

#!/bin/sh
# Chromium update script
# - by shad <shad@zaphod.eu>

# mac | linux | ...
OS=mac
LATEST=$(wget -q -O - http://build.chromium.org/buildbot/snapshots/chromium-rel-${OS}/LATEST)
INSTALL_DIR=/Applications
TMP="/tmp/update-chrome-$RANDOM"

(
mkdir $TMP ; cd $TMP
echo Download...
wget -q -O chrome.zip http://build.chromium.org/buildbot/snapshots/chromium-rel-${OS}/${LATEST}/chrome-${OS}.zip
if [ $? -ne 0 ] ; then
	echo Cannot update.
	exit 1
fi
echo Unzip...
unzip -qq chrome.zip
echo Copying...
rm -rf "${INSTALL_DIR}/Chromium.app"
mv chrome-$OS/Chromium.app "$INSTALL_DIR"
)

rm -rf $TMP

Then, set a crontab rule to choose the update interval ($> crontab -e). Mine runs every day at 12:00.

# Run Chromium update script every day at 12:00
00 12 * * * /Users/shad/mbin/chromium_update.sh

I hope it will be useful for someone as much as for me.

CirruxCache: speeds up your HTTP app using Google Appengine as a CDN

shad — Fri, 30 Oct 2009 22:05:00 +0100

It is a great moment, for the first time since I have started to work at Zoomorama, I have just released as open-source an important part of our server platform.

I previously explained how to use Google AppEngine as a Content Delivery Network (CDN). CirruxCache project concretizes this idea. I released the first version based as the one we use in production.

Here is the features it currently supports:

honor Cache-Control
cache TTL override
several POP (Point Of Presence) configuration mapped on custom base-url
ignore query string
POST forwarding
expired entries garbage collection
extensibility

CirruxCache is not documented at the moment even if you would be able to use it after reading the comments in the app.py file. I'll document this app in the next few days, but if you need more documentation, don't hesitate to contact me.

The project website.

Speed up HTTP delivering using Google AppEngine

shad — Thu, 30 Jul 2009 15:16:00 +0200

Google AppEngine provides an high-level cloud service which means that your application will be distributed automatically on top of the Google platform. All of your code will depends on the AppEngine SDK, so it could be risky to develop complex application on it.

I develop a webservice application for content delivering and content publishing at Zoomorama. We currently use Akamai CDN as a simple cache layer to improve data delivering accross the world. It is interesting for me to use AppEngine in the same way: without changing anything on my existing code base. I have found some posts on blogs dealing with this AppEngine usage, but they are not focused on dynamic HTTP caching like a real CDN.

Principle is very simple, all HTTP requests on my AppEngine application will be copied to the AppEngine Datastore. Moreover data which are delivered through AppEngine are cached by AppEngine servers. The code below is a tiny proof of concept:

# HTTP caching on Google App Engine
# - by shad <shad@zaphod.eu>
#
 
import web # webpy 0.3x
from google.appengine.ext import db
from google.appengine.api import urlfetch
 
origin = 'http://my.website.com'
 
urls = (
    '(/.*)', 'Root'
    )
 
class Cache(db.Model):
  data = db.BlobProperty(default=None)
  headers = db.ListProperty(str)
 
class Root(object):
  def GET(self, request):
    cache = self.readCache(request)
    if cache is None:
      cache = self.writeCache(request)
    for h in cache.headers:
      print h
    return cache.data
 
  def readCache(self, key):
    cache = cache = Cache.get_by_key_name(key)
    if cache:
      return cache
 
  def writeCache(self, request):
    url = origin + request
    response = urlfetch.Fetch(url=url)
    if response.status_code != 200:
      raise web.NotFound()
    cache = Cache(key_name=request)
    cache.data = db.Blob(response.content)
    cache.headers = []
    for k, v in response.headers.iteritems():
      cache.headers.append('%s: %s' % (k, v))
    cache.put()
    return cache
 
if __name__ == '__main__'
  app = web.application(urls, globals())
  app.cgirun()

I use webpy to depends on the AppEngine SDK as less as possible.

I have almost finished the production version of this application. I am doing some performance tests. This application is closed source for now. But I am going to release the code source in few weeks.

This version will include:

Fetch from Memcache (about 10 times faster).
Headers forwarding.
Read "Cache-Control" and "Expires" to define a TTL (rfc 2616).
Multi origins (according to url mount points).
Other small features (force TTL, ignore query string, etc...).

It is important to note that AppEngine does not keep running instances of your application (your CGI is distributed and it is executed on demand). So this application have to start very quickly (no configuration file, no dynamic generation, etc...).

Generates your C++ bindings easily with SWIG

shad — Wed, 01 Jul 2009 19:20:00 +0200

Today, I have to develop a client library. This client library aims to allow users to use my storage webservice app more easily. Mainly to avoid them to read my HTTP Rest API documentation.

I want to target the most used languages, and of course, I don't want to develop the same library for each languages. The main reason is that my client library is a draft for the moment, and I want to make changes on the client API over the time.

This library is very high level, so I take only the HTTP core class for the example:

#ifndef __HTTP_H__
#define __HTTP_H__
 
#include <iostream>
#include <fstream>
#include <string>
#include <map>
#include <vector>
#include <exception>
 
#include <curl/curl.h>
 
 
namespace HTTP
{
  class Error : public std::exception
  {
    private:
      std::string _message;
 
    public:
      Error(const std::string &msg) : _message("HTTPError: ")
      {
        _message.append(msg);
      }
      virtual const char* what() const throw()
      {
        return _message.c_str();
      }
      virtual ~Error() throw() {}
  };
 
  class Response
  {
    private:
      friend class Request;
      std::map<std::string, std::string> _headers;
      std::string  _version;
      std::string  _code;
      std::string  _message;
      std::string  _body;
 
      void  _parseHeaderLine(const std::string &headerData);
      Response() {}
 
    public:
      const std::map<std::string, std::string> &getHeaders() const { return _headers; }
      const std::string  &getVersion() const { return _version; }
      const std::string  &getCode() const { return _code; }
      const std::string  &getMessage() const { return _message; }
      const std::string  &getBody() const { return _body; }
      friend std::ostream&  operator<<(std::ostream &out, const Response &resp)
      {
        std::map<std::string, std::string>::const_iterator it, ite = resp._headers.end();
        for (it = resp._headers.begin(); it != ite; ++it)
          out << it->first << ": " << it->second << std::endl;
        return out;
      }
  };
 
  class Request
  {
    private:
      std::string      _host;
      unsigned int    _port;
      CURL        *_curl;
      struct curl_slist  *_headerList;
      Response      _response;
      std::ofstream    _outFile;
      std::ifstream    _inFile;
      std::string      _credentials;
      std::vector<char>  _buffer;
 
      void  _request(const std::string &uri);
 
    public:
      Request(const std::string &host, const unsigned int port = 80);
      virtual ~Request();
 
      virtual void  setHeader(const std::string &name, const std::string &value);
      virtual void  setCredentials(const std::string &login, const std::string &passwd);
      virtual void  get(const std::string &uri, const std::string &outFile = "");
      virtual void  post(const std::string &uri, const std::string &data);
      virtual void  postFile(const std::string &uri, const std::string &inFile);
      virtual void  setVerbose();
      virtual void  setHost(const std::string &host, const unsigned int port = 80);
      static size_t  writeHeaderCallback(void *ptr, size_t size, size_t nmemb, void *stream);
      static size_t  writeCallback(void *ptr, size_t size, size_t nmemb, void *stream);
      static size_t  readCallback(void *ptr, size_t size, size_t nmemb, void *stream);
      virtual const Response  &getResponse() const { return _response; }
  };
}
 
#endif /* !__HTTP_H__ */

SWIG uses interface files to parse and to generate bindings. IMHO, a good thing is to have an only one interface file to add language bindings much easier.

Here is some documentation on SWIG interface syntax.

First, here is my interface file (http.i):

%module http
 
%include stl.i
%include exception.i
 
%exception
{
    try
    {
        $function
    }
    catch (std::exception &e)
    {
        SWIG_exception(SWIG_RuntimeError, e.what());
    }
    catch (...)
    {
        SWIG_exception(SWIG_RuntimeError, "Unknown exception");
    }
}
 
%include "http.h"
%{
#include "http.h"
%}

All C++ code between %{ ... }% tags will be copied during generation. SWIG provides a lot of interface files. Some of them are language-dependant. So I have written a generic exception handler to use the same interface file for all targetted languages.

Finally, I just have to write a shell script to build each languages with the same http.i:

#!/bin/sh
 
CFLAGS_python_Darwin=-I/usr/include/python2.5
LDFLAGS_python_Darwin="-dynamiclib -lpython2.5 -o _zws.so"
 
CFLAGS_java_Darwin=-I/System/Library/Frameworks/JavaVM.framework/Headers
LDFLAGS_java_Darwin="-dynamiclib -o libzws.dylib"
 
CFLAGS_ruby_Darwin=-I/System/Library/Frameworks/Ruby.framework/Headers
LDFLAGS_ruby_Darwin="-dynamiclib -lruby -o zws.bundle"
 
CFLAGS_php_Darwin="$(php-config --includes)"
LDFLAGS_php_Darwin="-bundle -o zws.so"
 
if [ $# -lt 1 ] ; then
    echo Usage $0 language
    exit 1
fi
 
LANG=$1
ROOT=$(dirname $0)
OSNAME=$(uname -s)
ARCH=$(uname -p)
 
# All the source files
SRC="http.cpp"
 
(
DIR=${LANG}_${OSNAME}_${ARCH}
rm -rf ${DIR}/*
mkdir -p ${ROOT}/${DIR}
cd ${ROOT}/${DIR}
 
# This options will be always added to the compilation/linking
CFLAGS+="-undefined suppress -flat_namespace"
LDFLAGS+="-lcurl"
 
CFLAGS+=" $(eval echo \$CFLAGS_${LANG}_${OSNAME})"
LDFLAGS+=" $(eval echo \$LDFLAGS_${LANG}_${OSNAME})"
 
swig -I./src -${LANG} -c++ -o wrapper.cpp -outdir . ./http.i
g++ ${CFLAGS} ${SRC} wrapper.cpp ${LDFLAGS}
)

Notes:

./script.sh python to generate python bindings.
Some of the code below has been modified for the example, the most important thing to understand is the way of use.
I work on Mac OS X, so you just have to add (CFLAGS|LDFLAGS)_<lang>_<os> for other configs.
It would be cleaner with a Makefile

Cheers !

How to handle SSL certificates smartly in Firefox

shad — Fri, 03 Apr 2009 15:16:00 +0200

SSL certificates validation is often annoying. In most of cases, the certificate is not acceptable by Firefox because it is self-signed by the server. Some of personal blogs does not have their certificate signed by a known authority.

I think Perspectives is a good solution for this problem, which does not concern only Firefox.

According to perspectives website:
Perspectives is a new approach to help clients securely identify Internet servers in order to avoid "man-in-the-middle" attacks. Perspectives is simple and cheap compared to existing approaches because it automatically builds a robust database of network identities using lightweight network probing by "network notaries" located in multiple vantage points across the Internet.

Firefox extension: http://www.cs.cmu.edu/~perspectives/firefox.html

FreeBSD on the ASUS Eee Box using an USB flash drive

shad — Sun, 18 Jan 2009 20:28:00 +0100

I wanted my personal server to be closer to me. So I invested few days ago in the Asus Eee Box, thinking it is the best configuration for a light server at home.

I moved this blog and all my services from my dedicated server to my new one.

To setup FreeBSD on this kind of machine, you have two ways to do it:

Install FreeBSD using a pxeboot image and a dhcp server.
Install FreeBSD using a bootable USB flash drive.

Creating a bootable USB flash drive by using the original ISO image is a bit annoying if you haven't got a FreeBSD station yet. I have been using the fbsd2img.sh script,

I share my USB flash drive image: 7.1-RELEASE-i386-bootonly.img.
To setup the USB flash drive, use "dd" (/dev/XXX is the device of your USB flash drive):

dd if=7.1-RELEASE-i386-bootonly.img of=/dev/XXX bs=1m

Here is the result of the server hosting this blog

Subversion repository corruption: something to try when svnadmin recover does not work.

shad — Fri, 18 Jul 2008 11:04:00 +0200

I had the wrong habit to keep my old projects in a subversion repository. It's not the good way of keeping code safe.

svnadmin recover works when there is no corruption in your repository. If you have an old corrupted repository that you can't recover, you can first try a berkeley db recovery:

cd /var/svn/myrepos
dbX.X_recover

X.X is the berkelay db version of the repository.

If svnadmin recover and db_recover don't work, try the solution below, it worked for me.

You have to know the database version:

REPOS_PATH="/var/svn/myrepos"
DB_VERSION="4.3"
cd ${REPOS_PATH}/db
rm log.* __db.*
mv changes changes.old
mv copies copies.old
mv nodes nodes.old
mv representations representations.old
mv revisions revisions.old
mv strings strings.old
mv transactions transactions.old
mv uuids uuids.old
mv lock-tokens lock-tokens.old
mv locks locks.old
db${DB_VERSION}_dump changes.old | db${DB_VERSION}_load changes
db${DB_VERSION}_dump copies.old | db${DB_VERSION}_load copies
db${DB_VERSION}_dump nodes.old | db${DB_VERSION}_load nodes
db${DB_VERSION}_dump representations.old | db${DB_VERSION}_load representations 
db${DB_VERSION}_dump revisions.old | db${DB_VERSION}_load revisions
db${DB_VERSION}_dump strings.old | db${DB_VERSION}_load strings
db${DB_VERSION}_dump transactions.old | db${DB_VERSION}_load transactions
db${DB_VERSION}_dump uuids.old | db${DB_VERSION}_load uuids 
db${DB_VERSION}_dump lock-tokens.old | db${DB_VERSION}_load lock-tokens
db${DB_VERSION}_dump locks.old | db${DB_VERSION}_load locks
svnadmin recover ..

If binaries dbX.X_dump and dbX.X_load does not exist, you have to compile a old version of subversion with the berkeley database corresponding.

Thanks to dmp.

Link: http://svn.haxx.se/users/archive-2005-07/0576.shtml

Gmail-Notify improvements and a modified behavior

shad — Thu, 17 Jul 2008 14:21:00 +0200

Gmail-notify is a small program written in Python. I use it because it is light and it just do what I want.

I am a regular user of the firefox extension gmail-notifyer and I was uncomfortable with the Gmail-notify behavior, so I wrote a little patch to make it more like I want.

I also added the possibility to execute a command when a new message arrive. I use it to play a sound (aplay ~/sounds/mail.wav).

I don't know your habits but maybe you could enjoy my modifications. Here is a small description:

A click on the event box close it.
A click on tray icon open the mailbox.
Mailbox is now open as HTTPS.
A command can be executed when a new mail is received (like playing a sound).

To use my modifications, download my patch, and apply it to gmail-notify-1.6.1.1:

wget http://garr.dl.sourceforge.net/sourceforge/gmail-notify/gmail-notify-1.6.1.1.tar.gz
wget http://zaphod.eu/pub/gmail-notify-1.6.1.1.patch
tar zxvf gmail-notify-1.6.1.1.tar.gz
(cd gmail-notify ; patch -p1 < ../gmail-notify-1.6.1.1.patch)
rm gmail-notify-1.6.1.1.tar.gz

Then run gmail-notify.

gmail-notify/notifier.py

Ushare on Gentoo Linux

shad — Wed, 09 Jul 2008 00:01:00 +0200

Ushare is a UPnP media server on linux. I use it to browse media files from my Xbox. There is no official gentoo ebuild.

Sunrise Gentoo overlay provides an ebuild for Ushare, it requires some modifications to be used. I noticed some problems:

Ushare daemon is configured to run as root. Without being paranoid, I don't want to entrust a root daemon which manipulate my precious media directory.
Boot script in init.d directory is written for debian, so it does not work at all on Gentoo.
Default configuration file contains some wrong variable names. So this configuration directives are ignored.

I rewrote the init.d script to work on Gentoo and to launch ushare daemon as nobody user.

To sum up, Ushare can be deployed rapidly as follow:
1. Checkout Sunrise overlay with layman

emerge -v layman
layman -a sunrise
echo source /usr/portage/local/layman/make.conf >> /etc/make.conf
echo media-video/ushare >> /etc/portage/package.keywords
emerge -v ushare

2. Replace the Ushare boot script /etc/init.d/ushare by my script:

#!/sbin/runscript
#- by shad

DAEMON="/usr/bin/ushare"
NAME="ushare"
CONFIGFILE="/etc/ushare.conf"
OPTS="-f $CONFIGFILE"
PIDFILE="/var/run/ushare.pid"
USER="nobody"

[ -x $DAEMON ] || exit 0

start() {
        ebegin "Starting $NAME"
        start-stop-daemon --start --quiet --background --pidfile $PIDFILE \
                        --make-pidfile --exec /bin/su -- --command "$DAEMON $OPTS" $USER
        eend $?
}

stop() {
        ebegin "Stopping $NAME"
        start-stop-daemon --stop --quiet --pidfile $PIDFILE
        eend $?
}

3. The last thing to check is the name of configuration variables in /etc/ushare.conf. Make sure each variable starts with the "USHARE_" prefix.

Then,

/etc/init.d/ushare start

and watch your favorite movie on your Xbox 360

Windows Vista: BOOTMGR is missing!

shad — Mon, 04 Feb 2008 17:04:00 +0100

    I bought an additional hard drive last week to increase my storage capacity and to add a partition to backup data. I decided to move my Windows Vista installation to the second harddrive. Linux and grub are installed on the first one.

    Vista creates a Boot directory when it setups the MBR. I noticed that Vista does not create this directory when it is installed on the second harddrive, probably because the boot loader is on the first one. So after I restored grub to handle both windows and linux, when I tried to boot Vista, I saw the message "BOOTMGR is missing!".

    The boot directory cannot be seen on Windows systems, I verified its presence on my Linux system with the Windows partition mounted.

    I found a little trick to force windows Vista to create boot directory and bootmgr, I configured the bios to disable the first harddrive. Then I repaired the windows Vista installation with the dvd, the repair program created the boot directory and bootmgr program. Finally, I re-enabled the first harddrive.

    It's important to add a small setting to grub configuration to make Windows believe it is launched from the first harddrive. Grub is able to change the bios harddrives order virtually.

title=Windows Vista
rootnoverify (hd1,0)
map (hd0) (hd1)
map (hd1) (hd0)
chainloader +1

Duplicity: simple, efficient and secure backup solution for Unix

shad — Thu, 24 Jan 2008 19:51:00 +0100

Backuping data is a real problem. Investing 2 RAID 1 harddrives is a little expensive for a workstation. Duplicity is able to do incremental backups with the rsync algorithm. All backuped data are encrypted with gpg.

All duplicity options are available on the official website.

I wrote a small script for automate backup on my workstation.

backup.sh:

#!/bin/sh

ROOT="/root/backup"
DISTANT="ssh://user@zaphod.eu//home/user"
PASSPHRASE="xxxxxxxxxxxx"
KEEP_ARCHIVE="2M" # 2 Months
FILELIST="files.lst"

export PASSPHRASE
duplicity       --include-globbing-filelist "${ROOT}/${FILELIST}" \
                --exclude '**' \
                --full-if-older-than "${KEEP_ARCHIVE}" / "${DISTANT}"

duplicity remove-older-than ${KEEP_ARCHIVE} ${DISTANT}

Then a small file list specifies what has to be backuped or not.

files.lst:

- /home/shad/p0rn
- /home/shad/tmp
+ /etc
+ /home/shad

According to duplicity documentation, a line beginning with a '+' specify a directory or a file to include and '-' for exclude. Be aware of list order.

It remains to add a cron entry for a weekly schedule:

0 0 * * 1 backup.sh | mail -s "Backup Report" root@zaphod.eu

Now your worksation is secured, you can restore any files to any date within the backup delay.

Gnump3d: New release is imminent.

shad — Wed, 24 Oct 2007 17:30:00 +0000

Steve Kemp, gnump3d founder and current maintainer is going to release a new version of gnump3d.

This release will introduce some changes:

Password authentication is completely going away, due to some problems, especially discovery of a new hole which can be verified with netcat:

GET / HTTP/1.0
-> All looks good. You'll get a 403 header back.

Now try this malformed request instead:

GET /
-> Password auth bypassed

This hole can be explained by the home-made web server implemented.

"Split warning" will be removed when gnump3d is starting (index updating).
Some other bugs reported by mail will be fixed.

PwrCtl 0.2 released

shad — Sun, 15 Jul 2007 13:39:00 +0000

I have just release a new version of PwrCtl (0.2), some new features has been added :

Server notifying over UDP multicast: each server notify its presence on a multicast address to be displayed on the client interface (only available over Wifi, make sure multicast filtering is disable on your router).
Error handling improved: client is now able to auto-reconnect after a network problem.
Client session to keep context after reconnection: the client keep a session id, delivered by the server, to get back its context (profile, controller instance...)
Support of VLC and Windows Media Player for Windows.
Platform handling on server controller has been fixed.

You can get it now on download server.

All documentation has been updated on the wiki page.