SQL Server DB Mirroring Setup

Today I set up a SQL Server database mirror, complete with a third “witness” server that allows for automatic failover from the primary DB server to the mirrored hot spare.

Keep in mind all three machines need to run their SQL Service under accounts each server is aware of (create the same Windows account on each machine or just have them all on the same domain). The mirroring system effectively logs in to each machine’s SQL Service account to do its thing. it’s easiest to start out creating the primary DB, then doing a complete backup of both the data and the log, then restoring that DB (“with norecovery”) on the intended mirror machine, then right-click the primary DB > tasks > Mirroring and follow the directions (making sure you have already set up each SQL Server instance to run under the various accounts, as mentioned above).

Tomorrow I’m going to test out the performance impact of running the mirroring in High Safety mode vs. the higher-risk/higher-performance modes. I threw together a rather simple firehose-like C# app to throw data at the mirrored DB. I’m curious if and how I can break it.

Next up is to replace each machine with clusters.

Using Behavior-based Classifier Algorithms to Combat Exploitive Hackers

So, the phenomenally resourceful hacker/gold farmer community is quite the challenge. You’ve tried everything- chat filters, IP filters, email filters, captchas, mass banning of accounts… they always come back and within weeks it’s as if you had never done anything.

Ultimately, you have to go back to basics. What’s the real issue with hackers and exploitive gold farmers? It’s their behavior. They “break” the game, both immersion-wise as well as the economy. So, all you really care about is limiting that bad behavior. Really, you don’t care if they are in the game, even profiting from their activities… as long as their behavior is not beyond that somewhat fuzzy line of “appropriate and reasonable”. So, therein lies the answer. You have to target the behavior. After all, that’s what your GMs are doing when they patrol the servers.

The problem I’ve seen with most attempts at combating exploitive behaviors is that they end up figuratively building a wall. Walls only work when your enemies only walk on the ground and don’t dig under it, don’t utilize airplanes, don’t build ladders, and generally refrain from trying to solve the problem of the wall analytically. Dude, exploitive gold farmers and hackers have within their ranks some of the more intelligent and capable people in the world. In fact, I consider them a worthy adversary and deserving of honest-to-goodness respect.

The only possible “solution” will be in cleverly and aggressively building ever more general solutions to automatically classifying players based purely on their behavior. And this is something we can do. But it takes a lot of determination because it requires a lot of pieces, both technologically as well as inter-personally.

1. You need good hardware. Just go with the most bang for the buck. Focus on memory and lots of pretty fast, physically independent drives.
2. You need clever algorithms. Microsoft has a great start in this area, and it comes free with SQL Server- it’s called SQL Server Analysis Services. The 2008 versions are indeed better than previous ones.
3. You need some good TSQL skillz. This is fundamental. You can try to jump over this, but in the end you can’t get away from it.
4. You need some good organization of database tables as well as some good naming conventions (this is more important than it may initially seem, but you will be doing a lot of data shuffling, and tables and columns that have stupid-obvious names just feel good).
5. You need a good way to centralize large amounts of data. While you can directly query Production servers for the necessary data, extracting it all to a central server allows for a lot more possiblities, and you and your co-developers will learn all kinds of good stuff about your existing systems.
6. You need to have good friends working on the game itself. You will find all kinds of good things that need logging in order to allow for classifying player behavior.
7. You and your co-developers will have to figure out how best to streamline the process of identifying in-game events that need logging and then actually getting that logging set up. It’s never as easy as it first seems. If you can get this down, I guarantee it’s only a matter of time until the world is your oyster.
8. You will need to figure out how to best deliver polished reporting products. Again, Microsoft has done good stuff here, and it comes free with SQL Server. SQL Server Reporting Services will be your friend.

The pretty cool part about working on behavior-based classification systems is that you will really get to know your game well. That, and this stuff is pretty cool indeed.

Accounting Challenges Presented by Virtual Currencies

Say someone gets ahold of your credit card information. Then that person goes and opens up a Click and Buy account and funds that account with your credit card. In addition, he funds it with other people’s credit cards as well. Then he goes and spends that cash at 14 different online retailers. Then, 3 weeks later, after you notice a those fraudulent charges, you initiate chargebacks for each one. At around the same time a few other people whose credit cards were used to fund that Click and Buy account also initiate chargebacks. However, not every charge the fraudster made gets noticed and charged back. Some of the cash remains uncontested.

How does Click and Buy decide which stores to subsequently pass those chargebacks on to? Never mind that Click and Buy would likely see the pattern and simply reverse all charges from the obviously fraudulent account.

As it turns out, our friend the First In First Out accounting method comes to the fore. You have to do some scripting on top of very solid transaction logging tables, but it traces everything back as best as one possibly can. It’s still entails an assumption that cash is spent in the order in which it was deposited, so one can point it out it’s not perfect.

As a side note, if you’re developing a system like the above, do yourself a favor and maintain lookup tables for every column in every table… well, of course, only for those columns that don’t have, like, potentially unique values for each row. You can then simply point SQL Server Analysis Services (SSAS) right at those tables, and voila, you can make your reporting life easy as pie.. theoretically of course 😉

A Quote

Damion Schubert tossed out a great quote in the MUD-DEV list that to me encapsulates something that only an MMO can really deliver:

“Nothing your NPC
is going to say is going to be as interesting as the fact that your raid leader is cybering your guildmaster’s girlfriend, and everyone knows but him. Creating an interesting context to make the space sticky is great, but at the end of the day, an MMO’s real content is other people. If it’s NOT — well, then your MMO probably shoulda been a single player game. Woulda saved a lot of money.”

That’s why I want to figure out how to design mechanics that allow for emergent behavior systems rather than ones that define behavior. The kind of storytelling that will work in MMOs is not going to be like in the movies, or in books, or in choose your own adventures, or in tv shows, or in single-player games. Storytelling in MMOs will be like herding cats. At best. And on occasion, if not on all occassions, you’re going to find it better to change the story as it unfolds… keep changing things as players respond to the story… take that to the limit and you have what amounts to you actually being drawn into the in-game world and behaving as what amounts to an in-game faction or entity. Heck, let’s just embrace that and have several each GMs controlling some given faction of creatures or NPCs in the game. Better yet, let them be able to control members of their faction in a way that approaches a Real Time Strategy game, albeit at a slower pace. That could result in some good stuff.

Jury Duty

Well, I have jury duty possibly all this week and into next. It’s interesting and all thus far, but my overriding feeling is one of exasperation that we have to waste this much time and hassle into remediation of problems that didn’t need to be created in the first place. I can guarantee you the next time I’m about to commit something that could be construed as a felony I’ll think twice, if only for the poor jurors and the hassle my trial imposes on them 😉

Game Master Tools

In my ongoing experience as a Game Master for Knight Online I have found an ever-present challenge: that of a distinct lack of in-the-moment relevent tools.
By no means am I griping. It’s just this need for effective, relevent tools to facilitate the Game Masters’ roles of maintaining and enhancing the in-game play experience strikes me as an [other] opportunity to distinguish one’s game from the rest of the pack.
By “in-the-moment relevent tools” I am talking about tools to help GMs do things like identify patterns of player behavior so as to respond most effectively. I’m thinking of activities like tweaking tool interfaces so as to allow for the most efficient possible GM activities, creating new tools to help mine through various metrics to find useful tidbits, and working with the technical producer to identify what metrics would be most important to track.
The Game Master crew can be the best direct feedback mechanism for ongoing product development/fixes, so exceptional tools will likely enable [more] exceptional work. I think this may even be important enough that sacrificing GM staffing levels in favor of a full-time [web] developer devoted strictly to the development and maintenance of GM tools would yield a net profit of surprising levels.

Over its shelf life a game changes. The community changes, and the challenge is to keep the game relevent to the customers. It’s an ongoing challenge, and the Game Master crew is in the position to play a significant role.

That’s a thought that’s been simmering for a while now. For the time being I am flexing Excel and VBScript as much as I possibly can to make the tools I find I really ned on a day-to-day basis, but I am hitting the limit of what’s possible outside of more serious development work. I’m seeing opportunities for some really useful metrics creation that can be done without the need to hassle the game development staff. Sara Jensen would be proud 😉

User Generated Content… Game Industry Job too

Gamasutra had an interview with John Smedley posted here. I thought the following quote indicates SOE’s Free Realms will be a very good project to watch:

I rather give people some lines, and say “color inside these any way you want.” Then you can’t just generate whatever you want, but you have users creating content themed towards our goals.”

On a tangential front, I just got a job with K2 Network in Irvine. So, this means I’m moving from London to Irvine- I’m there now, in fact. I’m very excited, and I start work tomorrow!

Since it’s a Korean-developed game imported and translated to Western markets, I will be most interested to get some first-hand knowledge on the issues surrounding the Play for Free business models, cultural translations, and the art of operating an MMO (rather than producing it in the traditional sense).

I Need Context to Do My Job (or answer a question, or…)

When I was at GDC07 in San Francisco earlier this year, I landed an interview for a Design position that was opening up.  I spent hours writing out thoughts on as many issues as I could think of in an attempt to more precisely answer possible questions.

But, after all my preparation, the one thing the interviewer really focused on was the question “What would you do to balance our game?”  I briefly outlined a few general strategies, such as cash flow analysis using statistics and design goals, but for whatever reason I did not have an “ahah! moment” answer, which all the best answers are.  And, ever since, I’ve been pondering why a perfect answer didn’t click into place.

I know my lack of experience played a role.  But I think it was something more fundamental that I now know I needed: context.

In order to answer that question of how to balance the game, I think I really needed to spend a good several days in the staff, on site, talking with the lead designer, talking with programmers and scripters, looking at how the game systems operate, looking at how magic, buffs, defenses, damage, and speed work, and talking with the project lead and publisher.  At that point, I have absolutely now worries that I would know exactly how to balance the game.  Without that, my best answer can only be, “I’m not sure how I would balance your game.  I’d have to see it.  What have you already done in terms of balance?  What problems have you encountered?  How is your combat system set up?  Do you already have “reference point” items/loots/xp gains?”

I realize now that my answer should have been exactly what I felt it should have been: questions for him on the game, what they’ve done, where they’re stuck.  But instead, in my desire to impress, I succumbed to the worry there was already a canon of game balance techniques- and not knowing them by heart was a “wrong answer”-  so I tried to give a “right answer”.  And it went poorly.  Heh heh, they didn’t even give me a “thanks, but no thanks.”

I do not intend this to be a sob story.  I know I’ll be in game development some day, most likely the Producer side, but for now I just have loads to learn.  And one of them is that in order solve problems I need to know as much about everything as possible.  So if I ask lots of questions, please don’t hate me.

The Simple

I was reading Joe Ludwig’s Blog and his post Pizza Testing brought a smile to my face.  In short, to get great testing results, bring in a load of gamers, order pizza, and throw that whole mess at your game.  Then stand back, write down pertinent observations, and ask your “test subjects” to tell you what they are doing (or are trying to do).

 I guess I liked how very “human” that methodology is.  It’s a quick, dirty, yet very very effective solution.  Personally, I think it sounds like a blast.

Of note, Joe mentions the cost-benefit is such that he wishes they had been doing it long before.  That’s the part I like- the cost is so low that it is extremely doable.  In some ways I can imagine some amount of QA being offloaded to such sessions.  Or is such a suggestion anathema?

Wouldn’t it be Cool if…

I indulge:

I’ve noticed that in most every MMO I’ve played there is a sense that one can just go out, make some money, and then buy that leet Sword of Perth, or whatnot.  The important part is that being broke is never seen as a permanent thing.  Making money is assumed to be not only possible, but quite doable.  So….

Wouldn’t it be cool if real life laws and lawmakers (in the future) figured out what it was exactly that brings about this complete and utter economic optimism?

Obviously, most MMOs put the players in the situation of hunter-gatherers for the most part, that is, the “outside” world and the accompanying resources are limitless, and as a result wealth comes from outside the “village” (read player communities).  Whether or not real life is the same is arguable nowadays, since much value is found in what amounts to virtual assets- stocks, debt, services of other sorts.  But that’s a bit outside the scope of a “wouldn’t that be cool if…” musing.

This is where games, especially MMOs, will become of immense relevence.  Finding those players who are begging for in-game cash and figuring out what prevents them from seeing the opportunity in the game to make loads of it themselves… then finding the parallels in real life will be seriously useful.  The point is that I am confident the barriers to in-game thriving are going to parallel the barriers to real life thriving.

That’s why games matter.