Wardec is a sweet fun mechanic that CCP didn’t touch since the birth of Eve. What you have today is a universe for of corporation and alliances that can activate a wardec on anyone for any reason whatsoever. Until CCP does anything the aggressor has the advantage on the wardec system. Why ? That’s because they decide when they stop paying the war fee’s. Nice feature isn’t it ?

So your corporation or alliance has a wardec on your ass and you wonder how can you survive ? If your the CEO and don’t give a shit about it because you know, well pass this information to your members… they will need it. You never know!

Aggressor information

Who are they ? Let the officers or the CEO send information to the aggressor. Write them something to get information. The more info you have the better you can deal with this situation. You might try to send an eve-mail to try to negociate your way out of this  (highly improbable) but it’s free to ask and it never looks negative. It’s all about politics at this point. Hell, try to bribe him if you can.

Know the Aggressor

Use their killboard for a first look. You should know what ship they fight and how much numbers they use in their fleet, what ship are they using, what type of weapons are they using, do they use logistics and do they use electronic warfare ships or modules a lot ?

Go on the eve-forums and look for his corp or any member information in there. Try to google the info too (with corp or member name). The more information you have the better. When do most of the wardec pilots come online ?

Try to know how much you are dealing with too because only this will tell you if you can fight back or just… surrender.

Age of the corporation and/or the CEO

This is not a rule of survival but try to know how old and what is the ceo’s experience in pvp. This should tell you if he act’s like rambo or some dumb noob that is trying to prove himself.

Station games

Yes, the famous station games. If you don’t have a fleet ready to fight them, then DONT UNDOCK AND FIGHT. EVER. Seriously dont you dare undock. You will lose ships and you will encourage them to fight for weeks without stopping. What to do then ? Either stay docked and bore them to death or… Jump in another clone. If you do that, try to set another jump clone close to your location where you got pvp ships ready. Do and ask the same for your fleet and your corporation to do that as well. This way, you won’t be trapped in a station like idiots. Again, use jump clones as much as possible.

Systems to avoid

AVOID SOLAR SYSTEM MARKET PERIOD. Don’t go to Jita, Rens, Amarr, Dodixie… NEVER NEVER NEVER. Again, don’t go there. Anything with more than 150 people… AVOID IT. Most wardec whore will camp those systems and those stations because they know one vital information. Most pilots go in market system’s to get pvp ships. because of that, they camp those same stations you go too.

They are not stupid, most of the time, they scouted your corp or alliance first so they know where you go, where you hang and all. Avoid those system’s.

How to escape

Have you ever been trapped in a station before with some idiot camping it ? Use an undock bookmark. Just undock and don’t touch anything. Just activate your MWD or AB and bookmark the spot at 200km or more. If you didn’t touch the allign or anything that moves the ship, when you undock the next time, you will instantly warp to your safespot without getting locked. Well, less chance of getting locked anyway. It’s not 100% guaranteed but it’s better than nothing.

What ship to use, Use something that aligns fast. To hell with that shuttle. Personally, it’s better if you use a small frigate. Why ? Because you can fit a inertia stabilizers and some warp core stabilizers. With some defence modules you have more chance of survival. Good station campers will have an insta lock ready for you. Because of that, fast alignment and a “warp core stab” is the way to go. This should allow you go bypass gatecamp if they aren’t prepared for you little fast faggot ship. Hi hi.

Reschedule

Wardec is all about profit and killing shit. Nothing else counts. Deny those and you win the wardec even if you didn’t destroy 1 ship. You win only on principle. one way to do this is to long on at unusual hours. When your over with Eve-online, just dock in a station and let your character rot there. Just go to sleep or work and let the faggots camp your station. Station campers love to camp and do nothing. It’s there favorite thing to do in a wardec. I mean that’s why they use the war declaration mechanics in the first place ( /end sarcasm).

Neutral involvement

Use them for fuck sake. It’s legal, its a cheap tactic and it can piss of anyone who uses that method. You can get lots of smacktalk because of this.  What do I mean in case your not following ? Make another pilot use a logistics ship on yours. A pilot which is not in your corp or alliance. Once they remote repair you, they will become red to the enemy…that’s it and nothing more. Most of the time, they will not shoot back. If they do, use another neutral remote repair ship to repair the first one.

The fight

Well I will leave that one to you but DONT FIGHT ALONE. Ever. It’s plain stupid. Some nasty tactic is for lots of enemy pilots to log off in your system and once your aggressed (because you attacked an enemy ship which was alone) this lonesome idiot, he will tell ALL the others to log on. Once this is active, you will see a blob of ships targeting you fast. It’s legal and not considered an exploit so again, DON’T ATTACK ALONE. Use a fleet.

Have a neutral scout in neighbor systems and try to look for them if you want to attack them. if they are logged on, try to know where they are. If they are not… be on your guard. If it looks to good to be true then it is.

PS: I feel this guide incomplete but it should help you tremendously with your wardec. If you have any other advice, please use the comment section. I will add them to the guide later on.

Finally the results are in. Since the beginning of this year I’ve been waiting for those results. So much time has been taken to do those tests, well here they are.

Mass-Testing Results

It could be me but I feel that people are getting fed up with those test, lag problems and anything that is related to decrease their gaming experience down. I’ve read lots of thread on the Eve-Forums that they want more priority on fixing the lag and slow down problems – once and for all.

I understand CCP wants to release 2 expansions per year and want to keep it that way. For lots of reasons, its a good way to get people to play the game. They more people plays it the more they make money with it…logically. But they seem to forget something in this equation. The more people that plays, the more chance Eve-Online will be laggy.

Why ? With CCP’s current way is they give more priority to (future) expansions and less on the current mechanics and gameplay. If you don’t fix or improve your current gameplay, eventually, it will break. This is what’s happening right now. Lots of lag is invading our ships. The lag is so bad that it actually lagged the blog. “fixing lag: character nodes” got reposted again.

I don’t think CCP will lose lots of customer with their current way but they will lose their reputation. I remember not long ago, that CCP wanted to do incremental slow steady patches. This way, if something breaks, they could fix it fast. But it looks like they lost that touch. All I see (proof is in the patch notes) is a huge load of fixes. I know it’s officially written somewhere on the forum but you go take a look yourself.

Even though I don’t approve those new expansions (Incarna) I do welcome those mass testing events and those results. I do invite everyone to go on the sisi server when CCP ask for it.

First and foremost, there’s an update for Eve-Online on the 2nd of September. That’s because they discovered some issues. If you ask me they should post pone it even further to make sure nothing goes wrong.

Blog Lag

For some reason, CCP decided to post their “fixing lag: character nodes” blog again. Yes it’s a double post because it was published before. The odd thing is that they continue without fixing it. So my guess is they didn’t have any intention to post a new blog.

Surprise

To tell you directly, I got out of Faction Warfare. I had a big opportunity in null sec and I took it. I look forward to that adventure. For now, this blog will lay low for a bit. I will later decide what I want to do with it. I will probably continue to publish war stories and displaying guides but first I want to fix some things up before I continue with it.

See you soon.

It’s nice to see that CCP is keeping up to date with it’s programming language.  But when will they finally talk about content and gameplay ? It’s been a while since they haven’t talked about it. I’m not even talking about the new expansions, just pure simple Eve-Online gameplay. Well CCP did tell us they will post technical blogs so it should be over soon. After that they should begin talking about the new expansions.

From the lookout post of the impregnable CCP stronghold high in the Equus Mortuus mountains, we can see the world turning beneath us. As the universe slowly cools and we turn into ever older geezers, so also do the tools of our trade evolve.

As you may know, EVE has at its core the programming language known as Stackless Python.  When development of EVE was started, all we had to go by was a pocket watch, a piece of twisted wire, Stackless Python 1.5 and an LP with a band called Randver.  Stackless Python, back then, was centered around the concept of “Continuations” and it was rather tricky to use.

Later we moved to Stackless Python 2.0.  Along with the changes to Python itself, Stackless had grown the “Tasklets” and “Channels” that we have come to know and love.

We have since tried to keep abreast of developments in Python.  At various points in time we have upgraded our codebase to use Python 2.1, 2.3 and 2.5 successively.

Due to the amount of work that it entails, both the act of integration and the impact on developers, we have successfully dodged every other version.  But with each new version come benefits:

• New language features make development simpler and more effective
• The Python library is improved in terms of features and performance
• The language itself gains performance improvements.
• We stay current with a language in development.

Now, after a few years of sticking with Stackless Python 2.5, we have finally taken the leap and upgraded to 2.7.  Python 2.7 was released a few weeks ago, and subsequently, Stackless Python 2.7.

When we do an upgrade like this, it is not a simple task.  There is a lot of private modifications to Python that need to be migrated.  There is a lot of C code that needs recompiling.  And there is a lot of python code that may need some minor modifications.

For this reason, we are very careful to make sure that no unintended problems show up.  We run an extensive set of regression tests to ensure that everything works as before, and a full performance benchmark suite to verify that performance does not degrade.

As an example, we found during testing  that some client entry fields started working differently.  A user would enter a percentage, say 5.55%, and have it rounded to one fractional digit as 5.5% rather than the previous value of 5.6%.  It turned out that the rounding of floating point numbers had been subtly corrected in version 2.7 and that 5.5% is indeed the correct value (because 5.55 is actually stored as 5.54999).  To make such UI sensitive rounding cases work as expected switched to using the decimal module, a toolkit for doing arithmetic in binary coded decimal.

Python 2.7 is the last of a line.  There are no plans to continue with a python 2.8 version.  All Python development is now focused on the 3.2 version of Python.  But we won’t be going there any time soon.  Moving from version 2 to 3 is a much bigger leap:  There  are widespread incompatibilities on the API level and there are substantial language changes too. And at this time there appears to be  no immediate benefit to us in switching.

What effect will the upgrade have on the player?  In the short term probably none.  But in the longer term, it will make our job of providing quality services to you simpler, more enjoyable, and easier on our arthritic fingers and receding hairlines.

I’ve also got us a new stylus for our gramophone.  Stackless Python 2.7 and Randver will keep things moving for quite a while yet.

- Porkbelly

stackless python 2.7

CCP is doing well in fighting lag. I’m rather surprised by the amount of work they displayed in those blogs. I simply love it.  But still, 1.5m error in the month of June, didn’t they see this coming before June ?

As we’re all too painfully aware, when a few hundred pod pilots get together and exchange ammunition, the dreaded “lag monster” often comes around.  There are a number of distinct phenomena that are referred to as lag.  It is important to distinguish between them, as they typically have different causes/solutions (even if the symptoms seem the same, like “my guns didn’t work”). This blog is about the type called “module lag,” in which modules start to respond very poorly – sometimes for minutes late.  Specifically, this blog is about the bug that causes this for repeats/deactivations, and why it’s not fixed on Tranquility (TQ) yet.

At the Council of Stellar Management (CSM) summit in June, the CSM made an excellent presentation of the issues around fleet-fight lag, specifically about the issues of modules becoming ‘stuck’ and the workarounds used by players.  This provided useful insight into a problem which, until recently, we’d never been able to reproduce in a development environment.  This made the problem very tangible, and gave us a great symptom to begin digging into.

Like all the best bugs, this one has layers.  We’ll start at the first thing we noticed when digging at it:  the system responsible for telling the server when modules should be turned off or repeated would get minutes behind in processing when fleet fights happen while other systems remain reasonably responsive.  This system, named Dogma, handles module activation/repeat/deactivation, as well as the actual effects of those modules.  It also handles the various skill/ship modifiers you see in the game.

This sort of lopsided performance should be easily tweaked.  Tasks on the EVE servers use a time-sharing technique called cooperative multitasking which, in short, means that a task has to willingly yield execution to other tasks, otherwise it will run forever.  In this case, it would seem the part of Dogma handling module repeat and deactivation was being too nice – yielding execution too much.

Looking at the code some, a theory emerged as to why.  There was an error case that stuck out as odd – if an effect was supposed to be stopped or repeated, but the effect system itself didn’t agree that it was time yet, the code would throw up its hands and give up.  If that error case gets hit, the processing loop would yield to other systems early.  A code comment was very reassuring though – this error was supposedly “rare.”

The “rare” error happened 1.5 million times in the month of June, 2010 on TQ.

Above:  A re-enactment of my expression after learning this fact

So, there’s the first layer.  This error happens, Dogma doesn’t finish what it wanted to do (process every module event that should happen), the work load backs up and your modules start taking minutes to respond.  The typical thing to do here would be to run a test to verify this.  Unfortunately at that point the only environment we had to test on was the Singularity server mass tests.  So, off we go, putting in a setting to test to see if simply continuing past this error and finishing processing would be enough to alleviate the module repeat/deactivation problem.  There was a solid chance that it would.

Participants of the July 15th mass test: this is what I tested that made the node go horribly, horribly wrong.  Instead of clearing through the errors and charging on to victory, the effects processing loop instead choked on them, constantly hitting them and not yielding execution to anyone else until minutes later.  Interesting result.  The next mass test was three weeks away, so back to the lab I went to cook up something to test then.

And then, our savior arrived.  The Thin Client, as outlined in CCP Atropos’ blog .  These lovely little beasties, once properly tamed, allowed us to get a few hundred clients doing simple behaviors in the lab.  First, there were lazors:

Then there were drones:

Then there was orbiting:

And then there was lag!  The server I was running against was positively unhappy.  But still, modules were reasonably fine – the error was not to be seen.  (Remember what I said above about there being many manifestations of lag?  Well this was a different sort, but not what we needed to reproduce the ‘stuck module’ symptom)  Some fantastic sleuthing by my teammate CCP Masterplan lead us to some simple steps that could be done to induce this error.  Module delays quickly followed.

Once we have a reproduction in the lab that we can poke at, test new code on, and generally just play with, only the craziest bugs stand a chance.  This one took only a couple hours to pin down once this setup was in place.

So layer two!  The error, as you may recall from a few paragraphs up, was that Dogma was saying that the effect wasn’t ready to be stopped or repeated yet.  Well, why?  The calculation is as easy as you’d think it would be – start time plus duration.  It would take a lot to miscalculate that, so what the hell?

Turns out, the errors were coming about because the same effect was in the stop/repeat manager queue twice (or more).  One of the copies would process fine, and then all of the duplicates would generate the error because the effect was already repeated by the first one that processed fine.  So, duplicates in the manager caused the error, the error caused the process loop to yield early, the process loop yielding early caused module problems.

Next up, we put together a test to see if enforcing that there be no duplicates in the manager queue fixes the error, and therefore the module problem.  Cue the thin clients, they happily pew pew themselves for a few hours while we get the code right, and the result is a positive – fixing this layer does fix the symptom.

But good engineers don’t just fix a problem once, you fix it at as many layers as you can.  At this point, we’ve prevented duplicate effects from entering, but they shouldn’t be trying to get in there in the first place.  A quick trace dropped in where the duplicate is requested pointed us right at the top layer of this bug.  The code that’s responsible for making an entity (a drone, an NPC, things like that) go docile was calling the very same function that the stop/repeat manager was calling to stop its effects!  So specific timing cases, which are rare under normal circumstances but extremely common in high-load situations, were causing the effect to repeat instead of stop.  One quick tweak there and the top layer is fixed as well.

Thin client tests confirmed that fixing either layer fixes the symptom.  Cue the mass test!

The August 5th mass test had three rounds of combat in order to help us identify the impact of these bug fixes.  The first round had the fixes turned off, and the second with them turned on, no difference in style of fighting.  The fixes did exactly what they intended to do – repeating/disabling effects went from being 155 seconds behind at the height of round one to never getting more than seven seconds behind.  Success!

Well, almost.  As anyone who was at that test would tell you, the game was in bad, bad shape.  If you could get a lock and get a module to start repeating, it would repeat with gusto, but it rarely actually happened.  Locking took minutes, same with turning on or off a module.  Even warping away took a very long time.

There’s two problems evident from that test.  Firstly it’s clear that while relying on a bug to yield execution from this loop is a bad idea, not yielding at all isn’t a good plan either.  Secondly, and more importantly, reasonable numbers of pilots can generate enough load for the effects system that it can’t keep up.  That’s a dangerous game, taking  more than one second to process one second worth of effects – it quickly leads to doing nothing but process effects.  That’s what we started to see happen in the second phase of combat, which is scary business.

In the third phase we wanted to see where the tipping point was, so we started off with a decreased load by requesting no drones.  Everything was peachy, the system could keep up just fine.  Adding drones in doesn’t add very much load, but it was enough to tip the effects system over the edge of being able to keep up.  We were able to find where that tipping point was, and that will help guide what we do for TQ in the short term.

Essentially what we have to do is re-introduce yielding in the effect processing queue, only under our control instead of at the whim of some defect.  It will be after a configurable amount of time, and I’ll be present at some big fights to try to tune in on where that value should be to balance the needs of processing effects versus every other system.  Longer term, we’re working on some designs that could allow the effects system to keep up under very high load, which should finally put the nail in the coffin on module response issues.

In the meantime, we’ve been pointing the thin clients at various hotspots of performance and tackling them.  More on that as soon as the results are out for you to enjoy!

~CCP Veritas

PS – Big shout out to the CSM for their help to date pin pointing specific issues like this one and notifying us of fights as they happen so we can be there monitoring.  Also, thanks to all the players turning out for the Singularity mass-tests.  Much love~

Till then, toodles.

In the end, it’s suppose to mean less lag…long term I mean. But you will barely see the difference from what they say. Also, look at the comments, Dev’s are replies to lots of them. Some of them are very interesting too.

EVE has a deadly cobra strike force team alpha of extremely dedicated and proficient developers fighting the Lagmonster tooth and nail as their only mission. Through their work and the work of others, there’s now, at this moment, a perfect storm within the company and we have a great number of fixes in the pipes that will knock your socks off.

Our hope is that very soon our beloved Tranquility will be able to support fleet fights of a scale that far exceeds anything you’ve seen before, hopefully going beyond the roof of roughly one thousand on a dedicated node.

That’s tough talk, and we mean it. We will continue to go into detail in our ongoing series of dev blogs, some of which have previously been outlined in a dev blog by CCP Zulu. As soon as the optimizations are ready they will be pushed out to Tranquility individually and you will be able to gauge the difference yourself.

Character Nodes

In this blog I am going to talk about one of the optimizations we’ve been working on over the past few months: Character Nodes which we have already started deploying to Tranquility with phenomenal success.

The EVE Server is architected such a way that functionality is split into logical load balancing units and these units are statically assigned to a node which is a server process running on a single CPU core (since the server process is single-threaded). As long as any individual load balancing unit does not exceed the capacity of that CPU core we’re fine since we can just add more nodes if the load becomes too high. However, if a single load balancing unit needs to do more work than a single CPU core can handle then we’re in trouble.

A market region (Forge, Lonetrek, etc) is an example of a load balancing unit. We have multiple market regions living on a single node and currently four nodes servicing all the market regions. If the load on the market increases we can just increase the number of nodes dedicated to that task and decrease the number of markets on a given node. What this means is that when you’re in Jita and browsing the market, you’re not talking to the Jita node at all and don’t feel any Jita lag effects (up until the point where you buy or sell something in which case the Jita inventory system gets involved).

Another example of a load balancing unit is a solar system. Typically we will have multiple, even hundreds of solar systems living on a single node. We call these types of nodes Location Nodes. Typically these solar systems have such low load that they can be mapped onto a node with a lot of other systems in the same way as the market is. However, now comes the gotcha: If a single solar system exceeds the capacity of the CPU core we have lost the ability to further balance it.

This is the problem in solar systems like Jita and in systems where fleet fights are occurring. We cannot split the solar system up into more units and spread them out and we cannot spread the work out onto multiple cores. We are effectively stuck between a rock and a hard place (stupid GIL!).

Because of these absolute constraints it’s important that any work that doesn’t need to be done by the location node is taken elsewhere. Therefore things like planets (in Planetary Interaction), markets, corporations and alliances are load balanced separate of the solar system you’re in and if your EVE Client calls these services (such as when you are viewing your corp bulletins) then it’s talking to different node than your location node. Your client is at any one time talking to half a dozen different nodes, depending on the call context.

Because of the very strict request-response model that we employ (you click a button (make a request), the server does work, you get a response back) in our business logic a lot of the game systems lend themselves very well to distributed load balancing (e.g. away from your location). However, historically the location node has been viewed as your ‘primary’ node for a lot of the auxiliary logic that runs on the cluster. If a particular call doesn’t have a specific place to go to it will get routed to your location node. Now, that’s just lazy.

Today, after years of optimizations the only true remaining bottleneck on the tranquility cluster is the solar system location and we need to shave off every single cpu cycle that we can. With this in mind we introduced the concept of the Character Node in Tyrannis and have been moving services over to this paradigm since.

Figure 1: Node configuration

Figure 1 shows what this look like. Calls that would otherwise have gone to ‘Location’ are now routed to ‘Character’, which is a set of nodes that is very easily load balanced according to number of logged-in characters. This fits nicely with the schema that is already in place. We need roughly 8 character nodes (out of the total of 204 sol nodes in the cluster) to handle the load and this is very predictable.

Now that most of these changes have been deployed, when your client makes a call for things that aren’t directly related to your location, that call will typically be routed to your dedicated character node where your character happily ‘lives’ along with tens of thousands of other characters.

Since the work coming from an individual client has a long way to go before it exceeds the capacity of a single CPU core and is easily predictable, we’re in a good place. The client still makes the same number of calls to the cluster as before and the amount of work done on the cluster is unchanged. All that is different is that other nodes perform the work.

Changing this isn’t very glorious or filled with a lot of eureka! moments. It’s mostly just eating your vegetables and rearranging logic. The benefits, however, are substantial since, as it turned out, the majority of the calls that a client makes in a given period can be serviced by any node and therefore should not go to the location node.

The results

Okay, all the wall-of-text above was just to explain how the system works. Now that you’ve eaten your vegetables it’s time for the sugar-laden dessert. Let’s see the effect this had on Jita last week, comparing the performance before and after the phase #2 of these changes were deployed to Tranquility on 12 August.

Figure 2: Network traffic on the Jita node

In Figure 2 you can see the number of calls made onto the Jita node in four consecutive runs. After moving some services over to the character nodes, up to 80% of the calls were routed elsewhere freeing Jita up for important things like inventory operations and scams in local.

Figure 3: CPU utilization on the Jita node

As expected, moving services away from the Jita node had a good effect on CPU and has allowed us to scale Jita beyond the 1400 pilot limit that has capped its population for a while. Hopefully you should not be towed to another star system when you log in on Sunday evenings for a while.

Even though the metrics that we have gathered are for the Jita node, this change will have a positive effect on all loaded nodes in the cluster–with Jita and Fleet Fight nodes benefiting the most. The reason I use Jita here is that it has a very predictable load pattern whereas fleet fights are anything but. However, the same principles apply. Before this change you would be making something like 5-10 server calls to your location node to finish jumping, each one of these calls could take a long time to complete. Now you’ll be making something like 4, with the rest returning very quickly.

We’re hoping you’ll be able to tell the difference the next time you decide to invade your nearest friendly neighbor.

Also keep in mind that the other benefit of this type of offloading is that you don’t “feel” the lag as much. Take Jita for example. If you’re just browsing the market you don’t notice that it’s laggy since you’re not talking to that node, you’re talking to the market node. With the character node changes that we’re doing much of the user experience will be improved greatly since the buttons that you’re clicking end up on a lightly loaded node, even if clicking around in space is laggy. This can make a big difference in the overall playing experience.

We have been deploying the character node changes piecemeal to Tranquility throughout August and have a few more going out over the next few weeks. These should give you better performance in fleet fights as well as allow us to push Jita’s concurrent player boundaries further.

This is all well and good but keep in mind that ‘lag’ hasn’t been ‘fixed’ once and for all and it might not be in the foreseeable future, but we are pushing the boundaries of the cluster more aggressively these days than we have been for a while. Like I mentioned before, this is just one of the many things that we are working on right now. More services are being moved to character nodes this autumn and we will have plenty of more awesome fixes aimed at fleet fights coming in over the next weeks and months.

Fleet fight tests on Singularity

As stated before, it is very important for us to get as many people as possible involved when we are testing on the Singularity test server (SiSi). Please join in when the tests are advertised and help us test the performance improvements so that they can be deployed onto Tranquility as quickly as possible.

A word about fleet fight notifications

A fleet fight which happens on a node with a hundred other systems is not a good experience for anyone involved since often times the node only has 30% left of its CPU capacity when the fight starts. You can help make sure that the solar system you will be fighting in is on a dedicated node.

Use the Fleet Fight Notification form to let us know about a pending fleet fight. Sending in a petition is not the right way to report this, you must use the form. If you report the fight well in advance we can make sure that the fight is as smooth as we can make it.

Jon Bjarnason
Technical Director
EVE Online, CCP Games

fixing lag: character nodes

At last. The end with datacore gaining on inactive accounts. I hope people emo rage quit alone without going on forums to cry about it.

I like planning. I really do. I think calendars are pretty sexy and my OCD acts up when things aren’t color coded, have point values or do barrel rolls. This summer, I planned to sneak a few smaller items onto my teams backlog and positively surprise the Council of Stellar Management (CSM) when they got here. I ended up going to all these CSM summits, talking about features new and old, but never actually got to tell the CSM about some of the small things we had lined up. My plan didn’t work out, but the job got done, which is what matters.

Usually, smaller changes get bunched up and released with the coming expansion. This particular change is a little different though, as part of the code is needed for a bugfix, meaning we will be deploying it in the near future instead of this winter. The change we are introducing is going to end ghost datacore accumulation.

Currently, any character collecting research points from an agent will continue to do so, even after your account lapses due to inactivity. This is a pretty massive loophole to making substantial amounts of money and it is now being closed. When this change is deployed, characters will stop collecting research points when your account lapses into inactivity. You won’t lose the points you’ve accumulated up to that point, but simply won’t gain anymore. When you activate your account, your character will automatically begin earning RPs again. This change is long overdue and will hopefully benefit our active players.

On a sidenote, this was an issue our team picked off the CSMs list of priorities. It’s a great list that outlines both big and small concerns in the community and hopefully we can continue to address it. Summers are pretty quiet here, due to holidays and it’s a great time to go over the list and pick a few items for your team. Hopefully, we can bring you more items off the CSMs list, but till then, here’s one of them.

If anyone is interested, the CSM item is here.

The complete list is here.

Anyway, I should have more blogs coming up as we speed up for the winter release.

Till then, toodles.

the end to science from beyond the grave (and more)!

As mentioned previously in some blogs, I’ve been in faction warfare since September 2009. I’ve gone on breaks for some couple of months during that time but in all, I’ve been in it for quite long. Long enough to know that Caldari and Gallente throw knives at each other at every opportunity they have. Either verbally or with guns.

But in the last fleet I was in, the tears were EPIC. Yes the word epic is all in capital letters. It was that good. If I could compare the tears I saw I would ejaculate 5 times and even more. Yes, that was too much information. But since it’s my blog…go fuck yourself if you don’t like it LMAO. Experience the visual effect when you read my text, sometimes it can be very descriptive. I hope I wont have an ESRB rating squad on my blog, ah ah!

In this story, the war target fleet was in Kedama. Some of them were in Hirri. I was going to Nourv because invited myself in the fleet so that’s why I knew they were there. They jumped in Nourv just when I jumped too but since I was using a cov op ship, they couldn’t catch me.

When we formed fleet we had an advantage this time, we had some battleships and around 5 ecm ships. The enemy had mostly battlecruisers and lower. When we felt we were ready, we started to jump in Tama and started to lock the ships and kill them 1 by 1. From what I saw, half the fleet was there; not the whole fleet. Also as soon as they saw the small amount of battleships, they started to fled like cockroaches when someone turns on the light. Hi hi, I’m sorry, I loved that part. They jumped, warped away or just ran. I was very surprised ! I honestly thought they would fight like lions. In the end, we catched around 6 battlecruisers approximately . On our side, we did lose a falcon and someone said they lost their virginity.

The juicy part is next, just after the fight. Our FC received 2 hate mails and 1 private “convo” that told him to stop being a “bitch”. Here is the mail in question. This should tell you what this Gallente pilot thought of our FC:

From: Crying Gallente Pilot
Sent: tears sent after the fight
To: xxxxxx

Stop being such a child, man. Your fleet had 8 BSs just now. WHY are you not facing our battleships?

How many Scorpions you had there???

Grow up and come fight us. Or else I’m going to disclosure your childish fear to everyone.

Are you ready for EVE or not?

After the fight, we got word that they wanted revenge…badly! So they started to do a rather large fleet. Our Intel reported they saw 8 Navy Issue Dominix with lots of battlecruisers. In all, It was the same fleet composition but with 8 faction Domi added to the group. Unfortunately our fleet began to decrease because it was getting late so we decided to call it off.

When we tried to get more pilots in our fleet the local chat in “Nourv” was going crazy. Some locals didn’t like what we did so they started to use the same old smacktalk to defend what was left of their ego. Our FC lost control of his laughter when he saw Lion Silver repeated the same sentence “Gunny should stop playing eve” countless times over and over. Each time we said something he always replied “Gunny should stop playing eve” every single time.

This event was the most fun I had in a long time. I would like to congratulate the Gallente pilot who helped us have a really good time. Thank you guys.

You want to know about the “new” cool visual stuff in Eve. The new candy they are eating is awesome. But that women…no boobs, wtf ? hehe

Anyway, looks like this group is responsible for the cyno effect. From what I can see in the comments, they are looking for people that got skills in the visual effects and technical artists department. In other words, if you want a better cyno effect, apply, do a better cyno effect and shut the hell up about it.

Over the last few days you will have seen the blogs coming out from the boffins in our engineering teams about Mass Testing, Long Lag and Thin Clients. In some of these blogs you will have seen reference to ‘Core’ teams and so I wanted to take this chance to introduce the Core Technology Group (CTG), let you know what it is we do and where we sit within the CCP development structure.

At the end of June, our CEO Hilmar Veigar Petursson referred to CCP’s Core Technology Platform at his opening keynote of China GDC. For some time now, we have been internally branding the framework we use to build all of our games as “Carbon.”

Giving the framework a name helps us a lot with communication internally and of course to you, the wonderful players of EVE Online. There is also the old Icelandic saying of “if you know its name, you can kill it”, which basically talks about the power of knowing the names of things and how that gives you the ability to control said phenomena. As complex and multifaceted as our Carbon framework is then we certainly benefit from all the help we can get to wield it. And the custodians of Carbon are the Core Technology Group.

This video shows some of the Carbon technology in action as was presented by two CTG members at GDC this year.

<object width=”640″ height=”385″><param name=”movie” value=”http://www.youtube.com/v/Gf26ZhHz6uM&rel=0&border=1&color1=0x3a3a3a&color2=0×999999&hl=en_US&feature=player_embedded&fs=1″></param><param name=”allowFullScreen” value=”true”></param><param name=”allowScriptAccess” value=”always”></param><embed src=”http://www.youtube.com/v/Gf26ZhHz6uM&rel=0&border=1&color1=0x3a3a3a&color2=0×999999&hl=en_US&feature=player_embedded&fs=1″ type=”application/x-shockwave-flash” allowfullscreen=”true” allowScriptAccess=”always” width=”640″ height=”385″></embed></object>

So, why have Carbon and what is it really?

As CCP and EVE continue to grow, it makes sense for us to consider our existing technologies and re-use these if appropriate across our projects as part of the Carbon framework. This becomes more appealing if those technologies are proven (or, as we call it, ‘battletested’) in the crucible that is a game being used by a passionate, resourceful and sometimes devious player base. Furthermore, as our new projects continue to mature, we can take the technologies developed by them and apply them to EVE. This means that the EVE development teams get some great new technologies which it hasn’t had to spend any developer time creating. Having the CTG take the strain making this happen ensures the EVE developers can concentrate on developing EVE and make best use of this common technology.

It also makes sense for a central group (the CTG) to build brand new Carbon technologies which we know can be shared. This group can then deploy and support the technology across CCP to whoever needs or wants to use it.

So what is the Carbon technology framework?

Well, it is mostly things which operate at a low level in our games. The idea is that the framework will consist of all of the key technology pieces for an MMO that our game teams can pick up and use. The Trinity2 graphics engine was the first piece of Carbon technology (even if we didn’t know it as Carbon back then) introduced back in late 2007 by the newly formed Core group. Since then, various parts of our technology have been ‘Carbonised’ such as the graphics refresh that happened as part of Apocrypha. You can refer back to these Dev Blogs for details of things which have been made part of Carbon in the past.

StacklessIO

Core Tech – the beginning

Of course, just creating and deploying new or existing technology, even if it is ‘battletested’, is only part of the story. The CTG also spends a significant portion of its time re-working, re-writing and re-engineering parts of our codebase in a continuous effort to keep it up to date as technology advances. This also allows us to prepare the codebase to accept  new technologies into the Carbon framework, allowing us to keep pushing the boundaries of what EVE Online can be.

Who is in the Core Technology Group?

Well, the CTG started, as mentioned earlier, with a few graphics programmers working under the direction of our Chief Technology Officer to produce the next generation of the EVE graphics engine. From that small beginning, we have been hiring new people into CCP in order to fill a number of teams. The CTG is a separate group within CCP, it is not part of EVE. We do not count as part of the EVE headcount and we have a separate budget and hiring plan. However, in reality we work very closely with EVE although Core does not work on game features. We provide the reprocessed minerals that  the game teams then use to, in the case of EVE, build your serious internet spaceships.

We currently have 5 teams within the CTG, although we do use carefully chosen 3^rd parties when needed. Futuremark is a good example where we are working with them on a new part of the graphics rendering engine. It must be noted that whilst the CTG teams all work on Carbon, some parts of Carbon are not maintained by the CTG. These are usually specialist areas which can be managed by the team which has that expertise. Animation is a prime example. The animation team is based in Atlanta and the work they are doing will be part of Carbon.

The 5 Core Technology Group teams consist of the following…

2 x Core Graphics Teams (12 people):

These teams are those working in the bowels of Trinity2, making it perform better, provide sexier visuals and use more up-to-date technology. The Carbon Character Technology video from earlier in this blog demonstrates a small portion of their work-in-progress. As you can see, this team has been working on the graphics technology for avatars and the environments they will inhabit. For EVE that’s Incarna. These guys also develop and maintain our in-house tools system, ‘Jessica’, which is used by pretty much every developer in CCP and contains the Trinity2 rendering engine. As requests for improvements, new functionality or bug fixes come in from the EVE developers, the Core Graphics guys get on the case and deliver. Jessica is also used by the EVE video team in making all of our trailers, allowing them serious time-saving shortcuts in staging assets for dramatic narrative effect.

Core EVE Graphics Team (3 people):

This team consists of three Core Graphics developers who are 100% assigned to EVE, working as part of a nine person EVE development scrum. Due to the demands of working in a cross-project, multinational company, parts of the CTG must occasionally shift its focus onto a particular project on a temporary basis. When this concentration is not on EVE (currently it is), the Core EVE Graphics team provides the link and continuity of Core involvement in the graphics side of EVE. By staying 100% laser focused, they make sure EVE continues to get the Core graphics programmer support it needs. You may have seen the recent Dev Blog by CCP Blaze about Tyrannis Performance Improvements who is on this team. These guys are instrumental  in working closely with the EVE Art team to bring you new ships, planets and other graphical wonders.

Core Infrastructure Team (4 people):

This team is really at the heart of everything we are able to put out to our customers at CCP. These guys have created a common set of Productivity software, tools and technology which allows us to build more efficiently EVE and its patches from the source code, significantly reducing the time it takes to make EVE. This team has also built the repair tool which CCP Mandrake blogged about recently. In addition, this team has been working for a long time on delivering a way for our developers to be able to stress test EVE when they are developing new features or investigating and fixing bugs. We are now rolling this out and you can read more in the Dev Blog from CCP Atropos about the Thin Clients. I believe this is a significant step forward in how we are able to build and maintain EVE, allowing us to deliver a virtual world that we are much more confident is able to operate as we intend when we have record numbers of players hammering our servers and software.

Core Cluster Team (7 people)

What is one of the cornerstones of EVE? Lots of people interacting in a single universe without sharding. Obviously, EVE has scaled to dizzying heights over the last 7+ years and we now have more players than ever immersed in New Eden. However, we know that there are problems and we know we have to be constantly fighting to help the EVE cluster scale well beyond where it is now. This important task rests on many people all across CCP, from operations and virtual world staff to the EVE software developers to the EVE game designers and beyond. A key element in the fight to improve how we scale and reduce ‘Lag’ in the game is the work of the Core Cluster team. This team has the high level goal of developing our cluster technology to allow EVE to scale well into the future, even as we put more demands on it with more pilots inhabiting a more dynamic and developed EVE universe. This is no overnight fix and there have been some great discussions (as well as some painful ones) on EVE-O and other forums. Many of these threads have a good grasp, if not of the specifics, then of the general idea that we are trying to solve some of the hardest problems in a number of very complex disciplines. CCP Warlock works as the Distributed Systems Architect for CCP and the Core Cluster team and recently released an excellent blog on some of the challenges we are facing.

As recent blogs have started to describe, we are doing things right now to attack lag. We have people looking at specific bugs and issues which have been plaguing fleet fights. We are well aware that there are lag issues around large fleet fights (and some not so large ones) which are reducing your enjoyment of this part of EVE. We know because you have been telling us about your experiences regarding things like module lag, jump in lag and blackscreens to name just a few. We also know because we, as players ourselves (long time lurker, first time poster checking in after 5 years playing) experience these issues. I will leave the specifics of our progress against these problems to the actual developers working on “lag” who can get low down and dirty with some real techy blogs.

Genuine progress on these issues is being made thanks to the tools being produced by Core Infrastructure that are being used by the Core Cluster and EVE development teams. The nature of the lag problem means that it takes time to diagnose and address the problems, but we have invested significantly in getting the right tools and people to help us identify and improve the situation as fast as possible. Once fixes have been deployed to TQ and we have real evidence that they are making a positive difference to your playing experience, you will see more Dev Blogs detailing the processes and fixes.

So in terms of teams and what we work on, that is the Core Technology Group as it stands right now. You can find some more information here. We will continue to grow proportionately to CCP’s development teams and the growing scope of the Carbon framework. We are actively recruiting into the CTG and if you are interested, you can apply at the usual place, the CCP jobs page.

Now that you hopefully have a bit more clarity on how CCPs Core teams are structured, it is important that you, the EVE players, know that the focus of the Core Technology Group is on supporting EVE and the experience of playing it. Through our Carbon strategy we make sure that all future product development at CCP feeds back into Eve, as they all integrate back to Carbon. We also make sure that CCP has a robust battletested core framework to win our war against the impossible.

carbon and the core technology group

If a banana makes  a monkey happy this new thin client should make a developer happy. As long as I see sweat in a dev’s forehead, I’m cool with it.

This new client should help those Dev’s a lot by simulating Jita for example. If your not sure what I’m talking about, read on.

Over the last few months I’ve been working on something very cool that some of you may have heard about; it’s a project to rework the guts of the EVE game client to remove the audio and visual aspects of the game. In other words, to slim the client down as much as possible so that it can be considered ‘lite’ or ‘thin’.

And thus the Thin Client(TM) was born!

Click to see the thin client in all its glory…

What can this thin client do?

The basis for the thin client is the very EVE client you use yourselves; it takes that core and extends and overrides parts of it, so that you no longer need to have a sound card (insert generic EVE has sound meme) or a graphics card to run it.

Why should I care?

The thin client requires less system resources than a traditional ‘full fat’ client. As a result we can run more of them in parallel on one computer. Whereas a (normal?) EVE player might run 2 or maybe 3 accounts simultaneously with a traditional client, it’s possible to run many times this number with the thin clients.

The obvious benefit of a client like this is one of scale; we can start up many hundreds of these clients and have them do something, anything.

It now becomes possible to set them up so that we can undertake a controlled, large scale test; you can submit a new change to the code base and retest with the same setup to examine the effects of the code change. The level of control and precision these tests now give us is unprecedented.

Such practices have been used to load test websites for a long time, by repeatedly making requests to websites in an effort to discover the bottlenecks of the system. However, for EVE the closest we have gotten is the mass tests on Singularity that CCP Tanis runs.

The mass tests provide us with valuable data, but they can be very hard to exercise control over, since you are dealing with anywhere from 200 to 500 living breathing EVE players. The thin clients on the other hand are mindless automatons; if we say jump off a cliff, they will, metaphorically, go straight for the edge.

Ok, but what does this actually mean for me?

The thin clients themselves aren’t any smarter than a normal client. If you start up a normal EVE client it doesn’t suddenly start trying to take over the world ala SkyNet (hopefully), and the same is true of the thin client. To bridge this gap we’ve created a variety of methods through which we can tell the clients what to do: the two methods are internal projects called Orchestrator and the Automaton Project, both of which I’ll touch on later.

By being able to tell a client what to do we have created for ourselves a massive new tool box, which can be used to great effect. Allow me to elaborate:

1. It becomes possible to examine the behavior of massive amounts of mission running in the same system. To examine, as it were, the Rens Effect.
2. We can systematically examine the behavior of clients when they’re fighting large scale fleet fights, allowing us to recreate and diagnose the unique problems that large scale fleet fights create, in the lab.
3. We can place Jita under the microscope so that the impact of many thousands of market transactions can be understood in detail.
4. We can examine just why when one fleet jumps into another, the black screen of impending death appears along with more intricate reproduction steps beyond “get a big fleet and jump into another one”.
5. We can determine not only the theoretical threshold but also the actual performance threshold for fully loaded systems, whether it’s pilots idling in space, hunting NPC’s, shopping, afk-ing, anything.
6. And finally, we can evaluate the impact, at large scales, of new gameplay mechanics. Older players will recall many gameplay changes over the years, attributed to enhancing server performance such as the limiting of a ship to 5 drones from 10, changes to the rate of fire and damage modifiers on weapons to limit the impact high rate of fire weaponry would have on the server, for two obvious examples.

This new tool box allows us to load and stress test some of the oldest and most intricate components of the game.

Enough of the hurf blurf, give me the juicy stuff…

This is where I get technical, so if you fear techy, geeky nerd talk, skip this.

The obvious question is how did we achieve this? The core of the solution was through the application of two simple things:

• Mocking and mock objects
• Python class inheritance

For those of you with no programming knowledge I’ll clarify: mocking is the practice of replacing one object with another that is almost identical but allows a lot more control. It’s a process that is used in unit testing and allows the developer to test a piece of code in isolation. The use of mocking allows us to replace the pieces of the traditional codebase that rely upon a GUI with mock objects that do nothing. If you want to know more Wikipedia has a nice page on the subject.

The second step was the use of standard inheritance within Python to allow us to override particular pieces of the code; to explain I’ll run through a simple example:

Consider the targeting system: when you initiate a target lock on a ship, asteroid, or whatever, you’re telling the server to lock a target and to let you know when that is successful, or not, if they’re out of your range.

This is represented on your client as a new target appearing at the top of your screen. With the thin clients, we don’t have a GUI and so when the client gets the message from the server and attempts to load up the icon, it can go a bit haywire and raise errors about UI components missing and such.

By inheriting the class that handles the targeting, we can replace the single function causing the error with something that gracefully handles this new set of circumstances.

Of course, this can be very beneficial for us. It allows us to highlight areas of the codebase that are ripe for refactoring, where the game logic and the UI are too closely tied together.

In a lot of these cases, we’re reviewing and touching on older code and so we are getting ancillary benefits from reviewing these files from a more up-to-date viewpoint.

So what’s the performance like?

The average thin client has a memory footprint of between 150 to 200 MB. Now this may not be listed under the definition of ‘thin’ in everyone’s dictionary, but it’s a very good start. As we progress there will be more ways that we can reduce this footprint even more. As for CPU, the client requires very little; almost all the CPU required is in the first 30 seconds as all the Python libraries and code are loaded into memory. Once that’s complete the clients become relatively quiescent. Unfortunately when you run a few hundred of these at the same time, even minor CPU fluctuations, occurring across every client at the same time, can cause problems, so it’s something we’re keen to keep to a minimum.

But what about control? How do you tell them what to do?

Orchestrator is the framework that we’ve developed for running our system tests. Its primary function is to setup a server and client and to run a particular test on the client with traditional pass/fail mechanisms.

The only problem with this scenario is that Orchestrator is a very possessive system; it wants to have full control of everything, proxies, server and connecting clients, and for what we’re doing it proves a little too greedy. Because of the architecture of Orchestrator it’s not the ideal candidate for large scale control of clients, but it does allow us to run targeted tests making use of fewer slaved clients.

As for the Automaton Project, well, I have to point out, no one but me calls it that, it’s just my pet name for it. The project is a way of bootstrapping the client and having it execute arbitrary code locally, rather than having its movements dictated by a controller elsewhere on the network.

The difference between the two methodologies we have for controlling the clients is that one is a master/slave paradigm, whereas the other is group of fully autonomous actors; each has its pros and cons and we don’t want to blindly follow one particular path only to find that it’s actually the cause of our problems rather than the salvation.

So when can I get my hands on this?

Never, sorry The client is a developer tool only and whilst many people may want a less resource intensive client this isn’t the one you’re looking for </jedi>.

What now?

Now that we’ve got these tools, there’s work to be done creating tests for them. CCP Veritas has been toying around with them recently and has uncovered some interesting pointers whilst hunting the infamous lag monster, but I’m sure he’ll detail that in his own blog.

As for myself, there’s lots more API’s that need coding to allow the client to do more. Our primary goal is solving the lag issue, but beyond that I want to create a market interface that will allow us to setup mass trading so we can emulate Jita. There’s also turning a herd (flock? what do you call a group of these things? an army?) of asynchronous automatons into an organized fleet, then there’s work to be done on slimming them down further,  etc., etc., ad infinitum.

There is one thing I want to stress though: we still need your help. Once we’ve used these clients and other tools to track down problems and submit fixes, we’re going to need each and every one of you to lend us your time and effort to checking them on Singularity. EVE players have massive amounts of ingenuity, and we need to use that and resilience to help us stress test these fixes.

And on that note, au revoir!

fixing lag: and i, for one, welcome our new automaton overlords