Then all of a sudden, along came a post about how Amir Salihefendic had made a little daemon in C, using libevent and memcached, that would allow him to cache a http request in memcached. I don’t quite get why one would like to reinvent a reverse http proxy like that, and stated my arguments on why I thought that this seemed like a really bad idea.

As you can tell from the comments, he did not agree. It ended up with him deleting my last comment. Good thing I kept a copy in my clipboard. So, here are the comments.

Mads Sülau Jørgensen 25. Mar

This could just be me, but why would you do this over a proxy setup with Varnish, Squid og Apache with mpm_event and mod_cache? Seems a little like you’ve reinvented the wheel.

Also, as a previous commenter has suggested, why did you not just use the memcached support in nginx, and if it was because of the lack of multiple memcache servers, why not patch it and send this patch back to the community?

amix 25. Mar

Mads Sülau Jørgensen:

There are two reasons why I didn’t patch nginx with this support:

Separation of concerns: Running a separate polling server architecture I can monitor it and scale it independently Patching nginx would not have been that easy I prefer using memcached than Varnish or other caching options and memcached gives me a lot of options (deletion of multiple keys at once, consistent hashing and easy and proven scaling to terabytes of cache – - just to name a few).

Mads Sülau Jørgensen 25. Mar

Ah, but you could, easily, monitor a running nginx instance. The same goes for a running varnish instance. I’m not necessarily talking about running the nginx memcached module in the same process or the same server as your main proxy is running. So you’d archive the seperation by running another process or running the process on another box. Easy.

And the memcached module of nginx does not seem /that/ difficult to work with. Sure you’d have to do alot of handling of dead servers, and timeouts that could block the whole server. The current one probably handles this already. I don’t use it, so I’m not one to say.

With your keys structured correctly, you can flush (purge) multiple keys easily in varnish. You can provide your own hashing function as well. You could also use nginx’s hash balancer to send the request to the “correct” backend server.

The thing about hacking some server together with libevent in a language, that might not be onces main language is, that you’ll end up missing a lot of things, and it may end up exploding in your face. From the top of my head, what happens if a memcached server dies, memory leaks, what happens if the process segfaults (see memory leak).

It might be super-duper fast, but is it stable :)

amix 25. Mar

Mads Sülau Jørgensen:

plurk@web04:~$ ps aux | grep poll_server
plurk     29835  2.3  0.0   2376  1048 ?        
S    Mar24  27:23 /home/plurk/poll_server/poll_server 0.0.0.0 16000

It’s pretty stable (has not crashed since I started it), uses very little CPU and very little memory. If I stop the server, the nginx server will just route everything to Python servers. I doubt Varnish or patching Nginx would have been faster or more stable – - and they would have added a big conceptual complexity.

Personally, I would much rather fix errors in a 100 line program than a thousands line program. It’s separation of concerns and separation of complexity. So sure, I could have extended nginx+memcached or hacked Varnish – - but I chose to keep it simple, learn some new things and solve the problem in a very specialized way.

Mads Sülau Jørgensen 25. Mar

There is no hacking in the varnish solution what so ever. Varnish is a http proxy cache, it would be doing what is was designed to do. And my guess is, that it would be doing it well. My guess is also, that you’d end up using it alot, and to do more that using it with this one goal.

With a specialized solution, you have to handle a new process, you have to handle the errors that process can give, and you have to keep it running.

Using the tools already available makes more sense to me, and that’s what I’m doing when I run into a problem. Others, more proficient programmers usually have had the same problem before me.

amix 25. Mar

Mads Sülau Jørgensen:

Varnish is a good HTTP proxy and cache, no doubt about that, but it lacks a lot of things to be used in my setup. For example, how can I invalidate 1000 keys at once and do it in a split of a second? Their purging capabilities seem to be pretty primitive…

Plus, who said that I won’t iterate over this and go into a new direction – for example, benchmark this against a long-polling “comet” solution. Going into this direction would mean dropping Varnish – while I only need to change and add some lines of code to create a long polling solution of this.

Mads Sülau Jørgensen 25. Mar

Well, with a properly designed URL schema, you’d do something like this:

PURGE /poll/user/(?:mads|john|abc*)+/status

And then you’d have the VCL to handle that purge via purge_url(). The above will then purge the urls matching the regular expression, (so mads or john or starting with abc).

It actually works both fast, and well. It does not purge the cache when you tell it, but when you request an object. That makes invalidation fast.

See http://varnish.projects.linpro.no/ for more information.

You can more or less configure varnish to do what you want it to do via it’s configuration file.

As for the comet solution, your post more or less said that. Right about where it said something about push being hard to scale, which strikes me as a very odd comment. With push, you just have to be able to handle a horde of mostly idle connections, for which http://code.google.com/p/erlycomet/ or something like that, would probably be the way to go.

amix 25. Mar

Mads Sülau Jørgensen:

I am pretty tired of keep repeating myself. So this will be the last comment I answer you.

Varnish is a good cache, but it’s not really suited where I am headed. An example, how would you change the Varnish solution to support long polling? I can explore this option pretty easily with a basic understanding of how to build a custom server that can perform really good and that can be customized for my needs. I personally want to learn how to build things myself (also things that are complex and that require some thinking and engineering). I know I’ll need this skill longer down the road and practice makes a master – so the more practice I have, the better I’ll be at doing this.

this is the reply he deleted

> I am pretty tired of keep repeating myself. So this will be the last comment I answer you.

The consider this answer a bonus for your collection of entropy.

> An example, how would you change the Varnish solution to support long polling?

I’m quite sure that I never implied, that varnish would be suitable to do long polling (or other non-cachable long lived requests). If you think I did, please, tell me where.

> I can explore this option pretty easily with a basic understanding of how to build a custom server that can perform really good and that can be customized for my needs.

Yes, and that’s very fine and good, but that’s not really what your post says. It basiclly says “Using libmemcached and a C program, I can make a lot of http requests. Fast. And I don’t expect to use comet. Push is not what I want.” but from the looks of it, you may have wanted it to say “Hi, I made this little C program using libmemcached that I’m planing on developing into a comet server at some point. I need to extend the server with push in the future.”

> I personally want to learn how to build things myself (also things that are complex and that require some thinking and engineering).

Yes, throwing something together is very fun, throwing something together in C is even more fun. But throwing something together in C for a large website, because, it’s just fun, well, for a lot of the web developers in the world, that’s a really bad idea. It may work today, tomorrow but will it work in a week, a month or a year?

> I know I’ll need this skill longer down the road and practice makes a master – so the more practice I have, the better I’ll be at doing this.

Then I’d really recommend you looking at this:

http://aleccolocco.blogspot.com/2008/10/gazillion-user-comet-server-with.html

A comet server in C, with all the boiler plate completed. Just insert logic.

You might also get a kick out of reading up on memory management, input validation, how to avoid buffer overflows and just plain old K&R C.

As far as their interface goes, I can deal with that. It works, and it seems to do what it’s supposed to do fairly easy. A thing that has pushed me over the top, is the admins inability to help diagnostic a technical issue related to using their system as a secondary dns in front of a hidden primary.

Yesterday a good friend of mine told me that there was a new player in the dns game, one started by some people that seem to care about usability, exciting features (multiple dns templates per domain) and external api’s (goodbuy BeautifulSoup). It’s called QuickDNS.

So I rushed to their site, created myself a user and started tinkering around. Let’s just say, the it was actually a pleasant experience to do, things just worked^tm. So I setup a few templates and now have transfered a handful of my domains over there. Seems to work very well, just as expected.

So a very big congratulations to the whole QuickDNS team, you’ve done an excellent job, and made me a happy customer.

I seem to have agreed to give them a case of cola once they start implementing the axfr / hidden primary feature, and another one on completion. Can’t wait!

At work we only use open source software to create our solutions, and to run our hosting or these solutions. Amongst others we use Apache HTTPd, PostgreSQL and of course Debian GNU/Linux and I really love working with them. Now those I’ve just listed are quite mature projects and have a really large userbase. Apache in itself ofcouse is the de-facto HTTPd for any unix server, and it does it’s job incredibly well.

We also use a coule of unmature projects (in the term of lifespan, not stability), and amongst them are Varnish. In case you don’t know, varnish is a really, really high performance reverse proxy. It’s designed by the lovely red bearded man called Poul-Henning Kamp who is also a long time FreeBSD kernel comitter, which should say something about the quality of the code he writes.

To be quite honest, I don’t like the fact that Varnish has not made a release or a statement about it’s progress for almost 8 months. It should’nt be so hard, they have a trac, and some old readmap’s that are fairly outdated. Looking at the timeline of the trac tells us that there indeed is activity, but it does not tell us anything about the current status of the project.

The same almost goes for lighttpd, but they have gotten a bit better at communicating their status. It even seems they are planning to do a rewrite from the ground up in lighttpd 2.0.

What I’d really like to have varnish and the various other projects do, is keep an updated trac. I really fail to see how this whould not do both the projects and the community some real good. And for varnish, I’m inclined to do it myself should I be given permission and knowledge to do so.

Seeing as this is the thrid time we went down there in the last 3 weeks, we came up with a little ruleset for when we are allowed to ditch work, and have a relaxing cup of coffee.

Rules for café going for the Coniuro denmark office are:

• If it’s very hot outside
• If it’s very cold outside
• If it’s very windy outside
• If it became a habbit
• If it has been a while since the last time we went

If any of the rules should become true, we are allowed to go.

Having an office in the center of copenhagen has it’s advantages!

The pigeon’s of Copenhagen are quite lazy and probably won’t even move unless you are very close to stepping on them. You can easily walk very close to most of them, and you could hand feed them if you thought that was fun. I don’t. I’m more into stressing them.

How to play

The goal of the game is to make a pigeon take a long walk, the faster the better and the longer the better.

Rules are simple, 2 or more players can play at a time. It can be played by one self, but most people will just think you are an idiot, and probably give you money.

Rules

• No harming the little guy.
• No throwing objects.
• No interferring with the other players pigeon’s.
• When the bird starts flying or stops walking, you loose.

Now, go annoy some fat, lazy pigeon’s.