shad.cc

To content | To menu | To search

Tag - cirruxcache

Entries feed - Comments feed

Wednesday, July 21 2010

CirruxCache: Advanced configuration sample

By shad on Wednesday, July 21 2010, 22:46

That's it! This whole blog is cached and directly delivered by CirruxCache (only static files were cached before). My origin server is a tiny eeebox connected through my personal ISP.

So this configuration is a good challenge to offload my tiny web server as much as possible.

I think this is a good opportunity to show an example of a configuration which is a little bit more evolved.

The point is, I cannot set the same cache TTL for the whole website, and actually, I want to cache several websites...

# URL mapping
urls = {}
 
base = (
                '/_admin/(.*)', 'Admin',
                '/_store/(.*)', 'Store',
                '/_cron/(.*)', 'Cron'
                )
 
urls['default'] = base + (
                '(/debug/.*)', 'Debug',
                '/(.*)', 'Root'
                )
 
urls['www.shad.cc'] = base + (
                '(/themes/.*)', 'Blog_Static',
                '(/plugins/.*)', 'Blog_Static',
                '(/admin/.*)', 'Blog_Forward',
                '(/.*)', 'Blog_Page'
                )
 
urls['www.zaphod.eu'] = base + (
                '(/pub/.*)', 'Zaphod_Redirect',
                '(/.*)', 'Zaphod'
                )
 
# still supporting the old config
 
urls['cdn.shad.cc'] = base + (
                '/blog(/.*)', 'Blog_Static',
                '/(.*)', 'Root'
                )
 
urls['cdn.zaphod.eu'] = base + (
                '(/admin/.*)', 'Zaphod',
                '/(.*)', 'Root'
                )
 
# POP definition
# You can define and configure your Point Of Presence
 
class Blog_Static(cache.Service):
        origin = 'http://orig.shad.cc'
        forceTTL = 2592000 # 1 month
        ignoreQueryString = True
        forwardPost = False
        allowFlushFrom = ['x.x.x.x']
 
class Blog_Page(cache.Service):
        origin = 'http://orig.shad.cc'
        forceTTL = 3600 # 1 hour
        ignoreQueryString = True
        forwardPost = True
        allowFlushFrom = ['x.x.x.x']
 
class Blog_Forward(forward.Service):
        origin = 'http://orig.shad.cc'
 
class Zaphod(cache.Service):
        origin = 'http://orig.zaphod.eu'
        forceTTL = 2592000 # 1 month
        ignoreQueryString = True
        forwardPost = False
        allowFlushFrom = ['x.x.x.x']
 
class Zaphod_Redirect(redirect.Service):
        origin = 'http://zaphod.eu'
 
# !POP

I think this configuration is enough readable to avoid any explanation. However, do not hesitate to leave any comments.

Finally, I created a google groups to centralize all help requests. So if you need help, go to http://groups.google.com/group/cirr... or send an email to cirruxcache 'at' googlegroups 'dot' com.

Sunday, July 18 2010

New release: CirruxCache 0.3.1

By shad on Sunday, July 18 2010, 15:21

I am really glad to announce a new major release of CirruxCache.

This new release includes the following changes:

  • A storage webservice: store big files (<= 2GB) on the Blobstore in order to deliver them through CirruxCache. This feature is useful to bypass the 1MB limit on appengine.
  • An admin panel that enables users to flush objects, manage big files and see some statistics about the resources used.
  • Bugfixes

It is really important to note there are few limitations on the panel admin:

  • There is no error reporting on the flush panel (it only displays the number of objects trying to be flushed).
  • Storage manager displays a "500 Internal Error" when uploading. It only happens when you don't have a billing account (the Blobstore is only available on billing accounts, refer to appengine).

These two limitations will be improved in the next release, and there will be more informations in the statistics panel.

The Storage WebService will be documented really soon, but you can access the admin panel through "http://your.cirruxcache.app/_admin/"

I make the most of this opportunity to announce some changes on the project website:

I hope you will enjoy this new release.

2 comments

Wednesday, May 19 2010

Minor release: CirruxCache 0.2.2

By shad on Wednesday, May 19 2010, 17:45

CirruxCache 0.2.2 has just been released. It contains some bugfixes (thanks to Devattas to have reported errors on Datastore latency). Webpy has been updated to the last version.

I have also updated the documentation, especially I brought more details on Point of Presence configuration and usage of cron tasks for garbage collection.

Finally, some of the users reported me that there is a real problem with the cached object size limit (currently 1MB). I am working on the solution, I will take advantage of the new Blobstore service on AppEngine to store objects. Maybe I will keep the Datastore only for meta-data. This solution will raise the cache object limit to 50MB.

Stay tuned :)

2 comments

Thursday, March 11 2010

CirruxCache 0.2.1 is released

By shad on Thursday, March 11 2010, 17:20

I have just released a new version (0.2.1) of CirruxCache. To remember:

CirruxCache provides a software solution to dynamically cache HTTP objects on Google Appengine (using the Datastore and the Memcache services).

This new version includes an interesting set of features:

  • allow object flushing from restricted IP
  • configure a PoP (Point of Presence) according to a virtual host
  • several behaviors (cache, redirect, forward)

In more details, the last feature is the ability to configure a point of presence to differ from a classical caching mechanism. For example, I may want to configure "/admin/*" on my website to be redirected on the origin without caching.

Of course, this release includes several bugfixes, especially a fix on the "Expires" HTTP header which improves the caching performances.

Do not hesitate to test this new version and to comment any bugs or any suggestions.

3 comments

Friday, February 19 2010

Adding virtual host support to webpy

By shad on Friday, February 19 2010, 18:11

Webpy is a tiny web framework. I use it a lot for my web-services applications.

In general, I let my web server (lighttpd) to handle virtual hosting. But as you may know, I am working on a CDN solution on top of Google App Engine, named CirruxCache. In that case, while I have absolutely no control on the server configuration, I need to handle virtual hosting from the code.

Webpy maps urls by iterating through a tuple. So my solution is quite simple: wrapping the tuple to override the __iter__ function according to an environment variable (HTTP_HOST).

Let's take this basic webpy example, without vhosting:

import web
 
urls = ('/(.*)', 'hello')
 
class hello(object):
    def GET(self, name):
        if not name:
            name = 'World'
        return 'Hello, %s' % name
 
if __name__ == "__main__":
   app = web.application(urls, globals()) 
   app.run()

Let's add the VhostMapper class:

import web
 
urls = {
      'default' : ('/(.*)', 'hello'),
      'my-vhost.domain.tld' : ('/(.*)', 'helloVhost')
      }
 
class hello(object):
    def GET(self, name):
        if not name:
            name = 'World'
        return 'Hello, %s !' % name
 
class helloVhost(object):
    def GET(self, name):
        return 'Hello %s' % web.ctx.environ['HTTP_HOST']
 
class VhostMapper(object):
    def __iter__(self):
        url = urls['default']
        if 'HTTP_HOST' in web.ctx.environ:
            vhost = web.ctx.environ['HTTP_HOST']
            if vhost in urls:
                url = urls[vhost]
        return iter(url)
 
if __name__ == "__main__":
   app = web.application(VhostMapper(), globals()) 
   app.run()

Finally, you can use curl or wget to test your vhosts:

$> curl -H "Host: my-vhost.domain.tld" http://localhost:8080/

It is not so early to announce that the next version of CirruxCache will handle virtual hosting :)

I am sure this simple hack can be easily reproduced to use virtual hosting in some other Rest frameworks.

- page 1 of 2