shad.cc

To content | To menu | To search

Thursday, March 11 2010

CirruxCache 0.2.1 is released

I have just released a new version (0.2.1) of CirruxCache. To remember:

CirruxCache provides a software solution to dynamically cache HTTP objects on Google Appengine (using the Datastore and the Memcache services).

This new version includes an interesting set of features:

  • allow object flushing from restricted IP
  • configure a PoP (Point of Presence) according to a virtual host
  • several behaviors (cache, redirect, forward)

In more details, the last feature is the ability to configure a point of presence to differ from a classical caching mechanism. For example, I may want to configure "/admin/*" on my website to be redirected on the origin without caching.

Of course, this release includes several bugfixes, especially a fix on the "Expires" HTTP header which improves the caching performances.

Do not hesitate to test this new version and to comment any bugs or any suggestions.

Friday, October 30 2009

CirruxCache: speeds up your HTTP app using Google Appengine as a CDN

It is a great moment, for the first time since I have started to work at Zoomorama, I have just released as open-source an important part of our server platform.

I previously explained how to use Google AppEngine as a Content Delivery Network (CDN). CirruxCache project concretizes this idea. I released the first version based as the one we use in production.

Here is the features it currently supports:

  • honor Cache-Control
  • cache TTL override
  • several POP (Point Of Presence) configuration mapped on custom base-url
  • ignore query string
  • POST forwarding
  • expired entries garbage collection
  • extensibility


CirruxCache is not documented at the moment even if you would be able to use it after reading the comments in the app.py file. I'll document this app in the next few days, but if you need more documentation, don't hesitate to contact me.

The project website.

Thursday, July 30 2009

Speed up HTTP delivering using Google AppEngine

Google AppEngine provides an high-level cloud service which means that your application will be distributed automatically on top of the Google platform. All of your code will depends on the AppEngine SDK, so it could be risky to develop complex application on it.

I develop a webservice application for content delivering and content publishing at Zoomorama. We currently use Akamai CDN as a simple cache layer to improve data delivering accross the world. It is interesting for me to use AppEngine in the same way: without changing anything on my existing code base. I have found some posts on blogs dealing with this AppEngine usage, but they are not focused on dynamic HTTP caching like a real CDN.

Principle is very simple, all HTTP requests on my AppEngine application will be copied to the AppEngine Datastore. Moreover data which are delivered through AppEngine are cached by AppEngine servers. The code below is a tiny proof of concept:

# HTTP caching on Google App Engine
# - by shad <shad@zaphod.eu>
#
 
import web # webpy 0.3x
from google.appengine.ext import db
from google.appengine.api import urlfetch
 
origin = 'http://my.website.com'
 
urls = (
    '(/.*)', 'Root'
    )
 
class Cache(db.Model):
  data = db.BlobProperty(default=None)
  headers = db.ListProperty(str)
 
class Root(object):
  def GET(self, request):
    cache = self.readCache(request)
    if cache is None:
      cache = self.writeCache(request)
    for h in cache.headers:
      print h
    return cache.data
 
  def readCache(self, key):
    cache = cache = Cache.get_by_key_name(key)
    if cache:
      return cache
 
  def writeCache(self, request):
    url = origin + request
    response = urlfetch.Fetch(url=url)
    if response.status_code != 200:
      raise web.NotFound()
    cache = Cache(key_name=request)
    cache.data = db.Blob(response.content)
    cache.headers = []
    for k, v in response.headers.iteritems():
      cache.headers.append('%s: %s' % (k, v))
    cache.put()
    return cache
 
if __name__ == '__main__'
  app = web.application(urls, globals())
  app.cgirun()

I use webpy to depends on the AppEngine SDK as less as possible.

I have almost finished the production version of this application. I am doing some performance tests. This application is closed source for now. But I am going to release the code source in few weeks.

This version will include:

  • Fetch from Memcache (about 10 times faster).
  • Headers forwarding.
  • Read "Cache-Control" and "Expires" to define a TTL (rfc 2616).
  • Multi origins (according to url mount points).
  • Other small features (force TTL, ignore query string, etc...).


It is important to note that AppEngine does not keep running instances of your application (your CGI is distributed and it is executed on demand). So this application have to start very quickly (no configuration file, no dynamic generation, etc...).

Wednesday, July 1 2009

Generates your C++ bindings easily with SWIG

Today, I have to develop a client library. This client library aims to allow users to use my storage webservice app more easily. Mainly to avoid them to read my HTTP Rest API documentation.

I want to target the most used languages, and of course, I don't want to develop the same library for each languages. The main reason is that my client library is a draft for the moment, and I want to make changes on the client API over the time.

This library is very high level, so I take only the HTTP core class for the example:

#ifndef __HTTP_H__
#define __HTTP_H__
 
#include <iostream>
#include <fstream>
#include <string>
#include <map>
#include <vector>
#include <exception>
 
#include <curl/curl.h>
 
 
namespace HTTP
{
  class Error : public std::exception
  {
    private:
      std::string _message;
 
    public:
      Error(const std::string &msg) : _message("HTTPError: ")
      {
        _message.append(msg);
      }
      virtual const char* what() const throw()
      {
        return _message.c_str();
      }
      virtual ~Error() throw() {}
  };
 
  class Response
  {
    private:
      friend class Request;
      std::map<std::string, std::string> _headers;
      std::string  _version;
      std::string  _code;
      std::string  _message;
      std::string  _body;
 
      void  _parseHeaderLine(const std::string &headerData);
      Response() {}
 
    public:
      const std::map<std::string, std::string> &getHeaders() const { return _headers; }
      const std::string  &getVersion() const { return _version; }
      const std::string  &getCode() const { return _code; }
      const std::string  &getMessage() const { return _message; }
      const std::string  &getBody() const { return _body; }
      friend std::ostream&  operator<<(std::ostream &out, const Response &resp)
      {
        std::map<std::string, std::string>::const_iterator it, ite = resp._headers.end();
        for (it = resp._headers.begin(); it != ite; ++it)
          out << it->first << ": " << it->second << std::endl;
        return out;
      }
  };
 
  class Request
  {
    private:
      std::string      _host;
      unsigned int    _port;
      CURL        *_curl;
      struct curl_slist  *_headerList;
      Response      _response;
      std::ofstream    _outFile;
      std::ifstream    _inFile;
      std::string      _credentials;
      std::vector<char>  _buffer;
 
      void  _request(const std::string &uri);
 
    public:
      Request(const std::string &host, const unsigned int port = 80);
      virtual ~Request();
 
      virtual void  setHeader(const std::string &name, const std::string &value);
      virtual void  setCredentials(const std::string &login, const std::string &passwd);
      virtual void  get(const std::string &uri, const std::string &outFile = "");
      virtual void  post(const std::string &uri, const std::string &data);
      virtual void  postFile(const std::string &uri, const std::string &inFile);
      virtual void  setVerbose();
      virtual void  setHost(const std::string &host, const unsigned int port = 80);
      static size_t  writeHeaderCallback(void *ptr, size_t size, size_t nmemb, void *stream);
      static size_t  writeCallback(void *ptr, size_t size, size_t nmemb, void *stream);
      static size_t  readCallback(void *ptr, size_t size, size_t nmemb, void *stream);
      virtual const Response  &getResponse() const { return _response; }
  };
}
 
#endif /* !__HTTP_H__ */

SWIG uses interface files to parse and to generate bindings. IMHO, a good thing is to have an only one interface file to add language bindings much easier.

Here is some documentation on SWIG interface syntax.

First, here is my interface file (http.i):

%module http
 
%include stl.i
%include exception.i
 
%exception
{
    try
    {
        $function
    }
    catch (std::exception &e)
    {
        SWIG_exception(SWIG_RuntimeError, e.what());
    }
    catch (...)
    {
        SWIG_exception(SWIG_RuntimeError, "Unknown exception");
    }
}
 
%include "http.h"
%{
#include "http.h"
%}

All C++ code between %{ ... }% tags will be copied during generation. SWIG provides a lot of interface files. Some of them are language-dependant. So I have written a generic exception handler to use the same interface file for all targetted languages.

Finally, I just have to write a shell script to build each languages with the same http.i:

#!/bin/sh
 
CFLAGS_python_Darwin=-I/usr/include/python2.5
LDFLAGS_python_Darwin="-dynamiclib -lpython2.5 -o _zws.so"
 
CFLAGS_java_Darwin=-I/System/Library/Frameworks/JavaVM.framework/Headers
LDFLAGS_java_Darwin="-dynamiclib -o libzws.dylib"
 
CFLAGS_ruby_Darwin=-I/System/Library/Frameworks/Ruby.framework/Headers
LDFLAGS_ruby_Darwin="-dynamiclib -lruby -o zws.bundle"
 
CFLAGS_php_Darwin="$(php-config --includes)"
LDFLAGS_php_Darwin="-bundle -o zws.so"
 
if [ $# -lt 1 ] ; then
    echo Usage $0 language
    exit 1
fi
 
LANG=$1
ROOT=$(dirname $0)
OSNAME=$(uname -s)
ARCH=$(uname -p)
 
# All the source files
SRC="http.cpp"
 
(
DIR=${LANG}_${OSNAME}_${ARCH}
rm -rf ${DIR}/*
mkdir -p ${ROOT}/${DIR}
cd ${ROOT}/${DIR}
 
# This options will be always added to the compilation/linking
CFLAGS+="-undefined suppress -flat_namespace"
LDFLAGS+="-lcurl"
 
CFLAGS+=" $(eval echo \$CFLAGS_${LANG}_${OSNAME})"
LDFLAGS+=" $(eval echo \$LDFLAGS_${LANG}_${OSNAME})"
 
swig -I./src -${LANG} -c++ -o wrapper.cpp -outdir . ./http.i
g++ ${CFLAGS} ${SRC} wrapper.cpp ${LDFLAGS}
)

Notes:

  • ./script.sh python to generate python bindings.
  • Some of the code below has been modified for the example, the most important thing to understand is the way of use.
  • I work on Mac OS X, so you just have to add (CFLAGS|LDFLAGS)_<lang>_<os> for other configs.
  • It would be cleaner with a Makefile ;)

Cheers !

Wednesday, June 20 2007

HTTP digest authentication for gnump3d

Last month, I wrote a patch for gnump3d authentication, to add support of HTTP digest authentication. This patch has been included in the last release (2.9final). At present, this release is available in some linux distributions, but not in all of them (like gentoo).

If you need this feature, you will be able to find the patch here.

- page 1 of 2