The key considerations in any server architecture are maximizing performance, security, stability, scalability while minimizing implementation complexity. For any server application this is a delicate balancing act, but for a massively multiplayer game server it is a gargantuan undertaking. Creating a stable, high performance environment that can grow to support thousands of simultaneous players interacting in virtual real-time with anything approaching a reasonable level of development effort is no mean feat. In fact, more than a few experienced, professional software design firms have failed miserably in the attempt.

So the challenge is great. Great, but not insurmountable. I've spent a great deal of time over the last several years pondering this problem, forming and discarding more concepts than I care to count. Do I know the absolute, final answer? I'm certain I do not. In fact, I have no doubt that years from now I will still be contemplating this challenge. But I have come up with a few ideas I think are quite good, ideas which offer strong answers to all of the key concerns. And of all these ideas, one stands alone: the 5-tier queue-driven architecture.

Curious? Then read on! Oh, and if you aren't in to architectural stuff and you just want to get to the meat of the "why" question - you'll probably just want to skip straight over to the Conclusions section... Other than than, have a good read and I look forward to your comments.

Casey Dement

NOTE - This server concept is partially adapted from and greatly inspired by a message distrubution system I designed with considerably collaboration with Luke Kolin, one of the finer programming minds I've had the priveledge to work with. None of the concepts which drive this model could ever have been refined to the level of practicality they have achieved without his endless willingness to listen to my wild ideas and then puncture them with more holes than swiss cheese.

1. Client - This one is pretty obvious so I'll move on.
2. Gateway - This tier connects clients with the queue, handling all aspects of serialization and deserialization between network packets and queue events.
3. Queue - A logical [and/or physical] set of prioritized queues that give order to the application.
4. Process - Where all the actual thinking of the application occurs. Processes read events from the queue, execute upon the events, and then push resulting events back to the queue.
5. Persistence - And of course, this is where all of the simulation data is stored to / retrieved from.

The gateway tier can be thought of as the "public face" of the system. It can consist of one or more physical servers, load balanced through whatever scheme desired, which are the connection points for client sessions. It does not matter which gateway a client is connected to, and the death of a gateway server has no effect beyond disconnecting the set of client sessions which were attached to that box (which can be easily reconnected by the client).

The gateway servers also encapsulate all details related to the actual communication protocol between the client and the server, greatly simplifying the coding and readability of the rest of the server processes. This is accomplished through a translation layer between the physical packets and "system events" (as defined by the queue).

So what does this buy you?

• Performance - By limiting this tier to a very targetted set of tasks, we can get a whole lot of mileage from not much hardware. At least several hundred client connections per box should be no problem at all.
• Security - By unifying packet translation into a common model and hiding all business logic behind the public gateway, the vulnerability of the system to attack is greatly reduced. The more complex the system, the more chance of exploitation. Hardening a simple model like this, on the other hand, is child's play.
• Stability - Large, obvious stability gains come from this approach, both from the reduced complexity of the code AND by limiting the actual impact when a failure occurs.
• Scalability - This solution scales horizontally almost infinitely. An approach of this nature can easily handle thousands, tens of thousands, maybe even hundreds of thousands of simultaneous connections depending on hardware and infrastructure.
• Implementation Complexity - There is very little complexity going on here, thanks in large part to the very focused set of functionality this tier implements. The real volume of code really flows from the number of events that will be handled, but there is no additional bloat beyond the amount of code necessary to handle that number of events via any model.

The queue tier is both the simplest and the most important functionality of the entire architecture. It is also, being completely honest, the key single point of failure for the system. The queue, in a nutshell, is a prioritized collection of all events that currently need to be handled by any process (including the gateway tier) in the system. As processes are able, they poll the queue for the next task to be done. Once a process instance has retrieved a task, that task is removed from the queue so that no other process instance will duplicate the effort (and result). If, for any reason, the assigned process instance fails to complete the task in a reasonable time frame, that task is returned to the queue for the next available process instance. When a task is completed, it is permanently removed from the queue.

The most critical (and of course difficult) aspect of the queue functionality is maintaining synchronization between the vast number of process instances that are accessing the queue at any given time. As long as this consistency is maintained all is well with the server. So of course, it is VITAL that this code be as simple and stable as possible.

Also, it is important to note that there need not be just one queue. Since there are many types of events, many queues can be implemented on as granular an eventtype-by-eventtype basis as performance and hardware limitations demand. Just remember that unlike the other tiers, scaling in this way improves performance but does not improve redundancy. Each queue remains a single point of failure.

There are, thankfully, a great many very stable and effective ways to accomplish the needs of the queue. In past applications, I have even implemented the queue ENTIRELY at the database level, using connection-instance row level locking to maintain synchronization and automatic failure recovery. Such a simple approach could theoretically be appropriate for this application as well, but for several reasons I suspect a purpose-built solution is in order.

And of course, before we move on let's see how this aligns with our key considerations:

• Performance - Something of a trick question. The performance of the queue itself can obviously be tuned to very high metrics, but what about the effect of the queue layer on the performance of the system in general? Clearly, this layer of abstraction between the gateway and the process tiers increases processng latency. But is that the real story? More on this in the Conclusions section...
• Security - As mentioned on the gateway tier, everything past the gateway is completely isolated and abstracted away from any potential security vulnerability.
• Stability - Given the singularity of function and the obvious need to pay careful attention to this specific concern in the queue, excellent stability should be achievable without major effort. As a single point of failure, however, this tier is clearly less robust than the other tiers it sits between.
• Scalability - The queue, since it is not redundant in nature, cannot be scaled as far as the other tiers of the system without substantial additional work. That is not to say that it is not a scalable solution - even with meager hardware the system can easily scale into the "thousands" of users category. But "tens of thousands" is unlikely and "hundreds of thousands" is out of the question without expanding to a redundant two-phase commit model that would require far more latency tuning.
• Implementation Complexity - The price we pay for the simplicity the queue brings to the gateway and process tiers of the application is here, in the queue tier. Creating a stable, unbreakable synchronization and locking system is an incredibly difficult challenge. Fortunately, it's a challenge I've already had to meet and conquer in previous generations of this concept so we are in a strong position to execute effectively.

The process tier is a collection of seperate process servers that consume and produce the various types of events that occur in the simulation (how's that for a mouthful of not much help?). At a logical level each process can be thought of as an individual server which performs some task, and any task that must be performed is technically a seperate process (movement tracking, chat, combat, etc). On a physical level, however, each process is actually a set of one or more independent process servers independently competing to execute the next task of its type in the queue. Multiple processes (performance allowing) can also be combined on process servers as appropriate.

Fundamental to the process tier is the rule that all process servers must execute independently, with no regard for or knowledge of their peers. In this way, individual process servers can be added or removed from the cluster in real time with no interruption to the operation of the simulation. When an existing server goes away, any tasks it was executing are automatically returned to the event queue for consumption by another server. When a new server comes online, it begins consuming events. As long as at least one process server for each event type continues to function, the system need not go offline (though if sufficient infrastructure is lost obviously performance will be degraded).

And here is where the real payoff of this architecture becomes apparent:

• Performance - This model really blows the doors off other approaches in terms of performance. Each process can be implemented independently and optimized for its explicit needs.
• Security - This tier (all critical business logic of the simulation) is completely isolated from the outside world both physically and logically == essentially bulletproof.
• Stability - As with the gateway tier, maximum stability is achieved through the combination of isolating discrete logical functions for simplicity, seperating functions at the hardware level to reduce the impact of a failure, and the natural redundancy of parallel scaling.
• Scalability - Virtually unlimited, the only hard constraint is the infrastructure upon which the system is built (network bandwidth for example).
• Implementation Complexity - And again, there is very little complexity to be dealt with. Each individual process can be as granular and simplified as desired.

In all the details here I know it's easy to miss the big picture. So let me spell it out clearly - here's what we get from doing the server this way:

• The ability to scale the total bandwidth of the system in a near linear fashion to whatever level is desired - from a single server installation running a small personal server to a large cluster supporting thousands of players. This comes at a price - reducing the efficency of a single server installation. Then again, 2 small boxes are cheaper than 1 big one, so maybe even the little guys are better off...
• A very high degree of fault tolerance - in most cases failures (software or hardware) are absorbed without so much as a noticable hiccup to attached players. There are still single points of failure of course - but those points are isolated to keep them as simple and stable as possible. And, of course, all single points of failure can be recovered without compromising simulation consistency when a fault does occur.
• Reduced development effort, since most of the complexity inherent in a multi-threaed multi-user environment is hidden within the core architecture and game system developers need not know or care what the core does! The core necessarily is more complicated and requires more time and skill to build, but since 95%+ of the application code is writing all of the game systems that use the core the net gain is HUGE.

In a nutshell, I'm trading being able to do one thing really fast for being able to do lots of things fast enough - and doing so in a way that fundamentally enables almost everything to be done in a fully parallel redundant fashion. I don't know how to lay it out any more plainly than that.

So that's it - my rant is complete. Think I'm nuts - tell me! Got a better idea? Let's hear it!