My take on scaling

[fred]

In a separate thread Uday posted some details of a potential solution to our "scaling problems" that involved splitting BBO into 2 or more "mini-BBOs". He asked for feedback - thanks to everyone for your opinions.

 

The general sentiment was against the mini-BBO plan, but it seemed that some people were coming around at the end after some of the misconceptions were cleared up. I personally think the mini-BBO plan is much better than the alternative: changing the server archetecture to include multiple machines, but continue to tell each client everything that is happening on the system.

 

The purpose of this post is to try to justify my opinion. Comments are welcome.

 

Consider the following question: Why do people log in to BBO?

 

There are several possible reasons:

 

1) To play with random strangers in various places on the system (Main Bridge Club, any private/public club, team match, tournament)

 

2) To play with their friends in various places on the system.

 

3) To kibitz their friends and/or star players

 

4) To watch vugraph

 

5) To chat with their friends

 

6) To use any of the facilities available through Explore Bridge!

 

7) No specific plan (see who's online, do some clicking, join tables, leave tables, hang out...).

 

Consider how each of these fares with plan A (mini-BBOs), versus plan B (try to continue with one big BBO):

 

For 1), as long as each of the mini-BBOs has a sufficiently large population, it won't make a difference whether we use plan A or B.

 

There was a common misconception in the other thread that is relevant here. People seemed to think that we are about to split our, say, 5000-member BBO, into 10 500-member mini-BBOs.

 

That is not what we have in mind.

 

The idea is that, in the future, we may have 10,000 (or 20,000 or...) people online at once. If there were only 2 BBOs with 5000 people each, then the experience of trying to find a random game to play in would be no different than it is today.

 

Sure there might be some specialized smaller mini-BBOs for various groups, but we would aim to always have enough people logged in to the "generic mini-BBOs" such that it would always be relatively easy to find a pickup game.

 

2) We recognize that the whole "friends thing" is an integral part of the BBO experience for many of our members. If we went ahead with the mini-BBO plan, then we would certainly give people some easy mechanism for seeing which of their friends were logged in to which mini-BBOs, for chatting to their friends even across different mini-BBOs, and for joining their friends on a different mini-BBO.

 

I believe we can write a sufficient good interface to support these facilities such that there will be no loss of friend-related functionality if we go with Plan A instead of Plan B.

 

3) Similar argument to 2 above. We will make it easy for people to find which servers their friends and stars are playing at and to join those servers and get to the appropriate tables.

 

4) Vugraph would be shared across the mini-BBOs. There would be no loss of functionality in this area. We would have the option of making some vugraph broadcasts private for only specific mini-BBO(s).

 

5) See 2 above.

 

6) All BBOs would be able to access the same Explore Bridge! facilities. There would be no loss of functionality in this area. We would have the option of making some Bridge Library material (for example) private for only specific mini-BBO(s).

 

7) You should still be able to do all of these things just as effectively with the mini-BBO model that I have in mind.

 

Several people mentioned that the lobby list, as it is now, is too big to be useful. Some mention they would like to see only stars, friends, yellows, and/or people from certain countries displayed. I agree that these are real concerns and the View button in the lobby was created to help address these concerns. I imagine that most of you who made this point know about the View button. If not, please have a look. You can use the "save filter settings on exit" checkbox that appears in the window that you use to edit your profile so that you don't have to use the View button every time you log in.

 

In any case, managing huge lists will be less of a problem under Plan A than it would be under Plan B.

 

While tournaments rate to actually reside on a single mini-BBO, you will be able to see a complete list of tournaments and register for them regardless of the BBO that you happened to be logged into. I have no doubt that we can make this process seamless from the user's point of view.

 

In fact, the whole thing would be almost completely seamless. Aside from a new window that would do things like list available servers and which friends were logged in to each one, the only real difference with the mini-BBO model is that you would see smaller lists of people and tables (but we would try to make sure that these lists were always big enough so that people could quickly find pickup games).

 

So I think, if we do this right, there is almost no downside to the mini-BBO plan. I think I have a good picture in my head of how to do this right - sorry if I have not done a good job of explaining it.

 

Advantages of Plan A (mini-BBOs) include:

 

- Faster login times (MUCH faster in the long term)

 

- Ability to separate elements of our membership for various reasons. There are a lot of intriguing possibilities. For example, National Bridge Federations could license their own private BBOs for their members, or we could create a "BBO Hell" with no yellows and no abuse service and instead of barring people we could restrict them to logging in to our Hell server. We could create some kind of "premium BBO" that charges a membership fee and offer certain special services to only members of that BBO (say free weekly lectures from famous players). Some of these possibilities have the potential to bring in a lot of revenue for our company.

 

- Easier to implement than Plan B.

 

- Easy to strategically locate new servers in an attempt to help people in parts of the world where connectivity is often poor.

 

Hope I have managed to convince at least a few of you that the mini-BBO plan is the way to go!

 

Fred Gitelman

Bridge Base Inc.

www.bridgebase.com

[hrothgar]

Hi Fred

 

Your most recent post is covering an enormous amount of ground, touching on issues ranging from architectural, to social, to financial. I see a number of points that make a lot of sense. There are others that I'm less confident about. There are some that I find hysterical (In a good way... The notion of BBO "Hell" has already provided me with enormous amusement value. I think that I need to re-read the Divine Comedy for inspiration. We can create on Circle for those who willfully take advantage of UI, another for the wrathful, a third people who play Polish Club...)

 

Regardless, I think that discussions regarding mini-BBOs is premature.

 

From my perspective, mini-BBOs are best viewed as an implementation decision rather than a "feature" in and of themselves. As I noted in an earlier posting, I think that you would be best served by focusing on features that you wish to add to the software. Once you've defined a list of desirable features you can then determine an optimal implementation.

 

Case in point: Lets consider two specific examples from your most recent posting:

 

1. Is it worthwhile to provide private "playing areas" for different sponsoring organizations? For example, the ACBL might wish to create playing environment that mirrored face-to-face ACBL tournaments:

 

+ The official language is English

+ Alerts are based on the ACBL alert charts

+ Convention regulations are based on the ACBL's convention charts

+ Tournaments pay table fees to the ACBL and award ACBL master points

 

From my perspective this would be a valuable feature. (Then again, I was strongly in favor of creating "private clubs" which really didn't seem to go very far...)

 

2. Is it worthwhile to provide premium services such as a lecture series or tournaments that featured a high density of star calibre players.

 

Once again, I think that this would be a valuable service for the membership. Personally, I would prefer a model in which I was able to register for "events" on a case by case basis, however, I understand the attraction of a subscription based model.

 

If nothing else focusing on a list of features that you are providing to your customers rather than architecture should provide a better way to frame the issues being considered.

[hotShot]

If we can see our friends and can excange messages, join tables in a MBC and sign up for tourneys from all Mini-BBO's. I don't really care how you split the load.

 

But if I log in and can not find a tourney that i like to join, because it's running on a different Mini-BBO, or I can't watch my favorite player because i did not see (s)he's online on his Mini-BBO, miss a vugraph, because it's just transmitted in another Mini-BBO, i would not like that.

 

If we see our friends in our Mini-BBO Lobby list, even if they are logged a different Mini-BBO and if vugraphs and tourneys are reached without logging off and logging on again. I can't see the difference between "subfunctional server" (e.g. for vugraph and tourneys) and walking seamlessly beween mini-BBO's.

 

Yahoo has voice chat rooms, that can take about 25 people. If you join as 26th, you are alone in room 2.

 

If there is a second Mini-BBO, where everybody is lead to, if Mini-BBO one has reached 4000 player, than I might be alone in Mini-BBO 2 (not for long i bet). If you move player to balance this, we are back to what a load balancing system does.

 

Of cause i could understand, if privat clubs or NBO's or some volunteers would want to run a BBO of their own. This is something else. I don't know if i would want to join a NBO BBO server, because they might enforce their local rules and bidding systems, but there might be lots of people who would want just that.

 

But will it be the same e.g. if there is a BIL-BBO? If I enlist myself to a tourneys partnership desk now, stating "I'll play with BIL-Members" I'll get to know someone new. Will I have to log in to the BIL-BBO in future to do just that?

[david_c]

Several people mentioned that the lobby list, as it is now, is too big to be useful.

The length of the lobby list isn't really a problem. Most of the time, the default filtering (which pushes friends to the front) is adequate, and if you want some other filtering that's easy as well.

 

What is a problem is if providing everyone with the lobby list is a big contribution to the load on the system. If that is the case, then you need to seriously consider not providing this information automatically. The point is, if at some point I want to find out the complete list, I won't mind waiting a little while for the information. The same could be said about many other things - there's no need to know about the state of the main bridge club unless you're in the main bridge club (or you specifically request the information, in which case you won't mind waiting). Notice that this concept is almost equivalent to the idea of splitting BBO by function.

 

(Oh dear - now that I've written that, I can see it's been said many times before. Never mind, another voice can't hurt. Sorry to bore you all!)

[MickyB]

Hey Fred

 

In response to your comparisons between plan A and plan B...

 

2) I imagine it would still be a pain to get several friends to meet up on the same BBO.

 

5) Presumably it wouldn't be possible to chat to someone on a different BBO? In that case, if I want to talk to several different people I would need to get them to move to the same BBO.

 

Regarding the 'view' list - I doubt many are interested in filtering friends, yellows or to a lesser extent stars; Could we have the country and skill filters just applying to neutrals? IMO The only question is whether selecting 'experts' and 'US' should show all experts and all Americans, or just US experts. A box to swap between the two shouldn't add too much complexity.

 

While I'm not sure about the idea of a "BBO hell", it might be worth splitting those that tend to form friendships from those who play with randoms. In particular, could Hrothgar's "Permanent Floating Indy" be accessible from the one and only BBO, but joining it seamlessly move you to a new server? Manually run tournaments and team matches could move you onto another server. Like Hrothgar, I would rather that premium events are paid for individually.

 

It is difficult to know what to suggest without a greater understanding of how things would be different under this plan. Many seem to think that keeping the list of people online updated is the problem, but if that is so then I don't understand how the mini BBO plan would help without depriving you of the list of your online friends on other servers.

[luke warm]

i'm an end user, so i have no idea of the complexities involved... how, for example, does irc do it? iow, on most irc channels the room list doesn't load automatically, you have to invoke it.. don't know if that's possible on bbo, but it is a possible solution re: lobby

 

as for splitting, how would it work? will people have a choice of which bbo to log on to? right now i log on almost exclusively for reasons 2, 3, & 4... however, if bbo is split it might be difficult to do one or the other without logging off and back on to the *other* bbo (assuming there's a choice)

 

forgive me while i digress (regress?) to my zone days... the zone was divided into social and competitive rooms, and both were usually pretty full... is it possible to split bbo along those lines? the main diff i see is, on bbo more people (as a %) are more serious about their bridge.. fewer pure social players, iow

 

i guess i'm more worried about finding who i want when i log on, and about some friends on bbo#1 and others on bbo#2... for that reason i expect that most of the really good players will usually be found on one or the other, almost exclusively

 

is it possible to "split" from within one superstructure? iow, log on to bbo, and then choose a "room" ... the very choosing of the room takes one to one or another 'bbo'

[TimG]

2) We recognize that the whole "friends thing" is an integral part of the BBO experience for many of our members. If we went ahead with the mini-BBO plan, then we would certainly give people some easy mechanism for seeing which of their friends were logged in to which mini-BBOs, for chatting to their friends even across different mini-BBOs, and for joining their friends on a different mini-BBO.

 

4) Vugraph would be shared across the mini-BBOs. There would be no loss of functionality in this area. We would have the option of making some vugraph broadcasts private for only specific mini-BBO(s).

 

While tournaments rate to actually reside on a single mini-BBO, you will be able to see a complete list of tournaments and register for them regardless of the BBO that you happened to be logged into. I have no doubt that we can make this process seamless from the user's point of view.

It sounds to me like you think you can split into mini-BBOs without the user ever being aware. If that's the case: what's the problem?

 

Tim

[QuiteACard]

Following up on TimG's comment: would it be possbile to make the new BBO under plan A (i.e. mini-BBO's) completely "seamless" from the player's perspective? In other words present it to us as if only one physical server existed?

 

In this case we would never need a window that identifies which servers exist, which servers are available for use, which server we are playing on, and which "services" (e.g. tournaments, vugraphs, bidding practice) are running on which servers.

 

That give the administrators flexibility to physically organize the servers and to logically organize the "services" to meet BBO's growing needs and simultaneously simplify life for the players.

[DrTodd13]

If this were what they wanted to do then they wouldn't even tell us because it would be completely transparent to us. Yes, transparency should be a goal but before we go off designing a solution, let's decide first what our goals are; second, what the current problems are; and finally third, what the solution is. We can divide goals up into "must haves" and "nice to haves."

 

Must Haves

--------------

1. Must see all my friends who are online (whether they are on the same mini-BBO or otherwise.)

2. See all open tables.

3. Be able to request a seat at one of those tables with a single click (i.e., don't make me change explicitly my mini-BBO or something first in order to sit there.)

4. Be able to see which vugraphs going on.

5. Be able to see all the tournaments.

6. Find the tables where my friends or stars are playing.

7. Same as #3 except for tournaments and vugraphs.

8. Summary of the above is transparency.

 

Nice to Have

---------------

1. See everyone who is on-line.

2. See all tables.

 

Look at Yahoo messenger. There are millions of users and hundreds of chat rooms. I can add people to my friend's list and I am told when they come on-line. I can join chat rooms or send chats to my friends and it is all seamless. I don't see why many BBO functions can't mirror the scalability that Yahoo displays. What seems to be a scalability killer is for things like broadcasts (e.g., lobby chatter). I didn't put lobby chat on my wish list at all. Serving an individual table is inherently distributable. You could have clients do it or you could have any number of BBO servers handling table hosting responsibilities. We need to know Fred and Uday's goals as well. Are they network constrained. Are they machine constrained? Do they not want to buy/use more machines or more bandwidth. If it gets to a situation where it is a matter of programmer expediency versus not satisfying the user's wish list then people will take what they can get...after all it is free...but they may not be happy with it.

 

Personally, I think a lot of the work could be pushed to the client side and this would reduce server load and network utilization. Get rid of constant updating of the user/table/tournament list and replace it with "refresh when requested" and you've added a lot of scalability right there. I realize that all the friend stuff is handled locally by the client and that to fix this the code will have to change. Friends are going to have to be known by the server and an inverse map kept for each player (i.e., which players has listed this player as a friend). Then, when someone logs on, you can send a message only to those other players currently logged on who've listed this person as a friend.

[Resaz0]

Good morning,

 

I can remember that on Zone we used to have several rooms but most of them were always empty and all the people was connected on a few rooms that used to be full all the time. This happened because it was almost impossible to find a nice table in the "empty" rooms, as nobody was there.

 

This means that when you lose connection you have to make several attempts to find an "hole" and go back to your table, and people at the table has to wait a lot of time for the player who has lost connection.

It was extremely annoying.

 

I hope this is not going to happen again, I am playing only in team matches and such a behaviour of the system would be horrible, as finding a sub is always very painful (we do not like random sub players).

 

A part this problem I dont care of multiple bbo if I can see and chat with all my friends.

 

Have a nice day

 

Patrizia

[Rebound]

A bit of pie in the sky - while I understand your points, Fred, I just wish there was a way to re-distribute server load transparently so that we could keep our nice, familiar, easy to use BBO :)

 

My own only real concern is that of most people going into one particular BBO anyway because that is where most of the top players are.

 

IMHO a bigger problem may be the speed/quality of the internet connections of the users. I suggest some method of choosing a "lower bandwith" option at the client side, requesting less information be sent by default. I rarely have difficulties of any kind with BBO, but I am lucky to live in Canada, where internet connectivity is second to none. There are many parts of the world where this is not the case, of course.This is sort of off topic, perhaps, but I certainly agree that BBO servers distributed around the globe would help to alleviate this problem.

 

To be honest, it sounds to me like you have found a workable solution to a difficult problem.

[uday]

BBO is currently a single large community.

 

1. One thought we had was to split bbo up into multiple independent BBO communities just so that each community is of a manageable size.

 

2. Another was to simply deal with it, and allow bbo to grow as it will in a single large community.

 

 

It struck us that we could avoid much of the pain of (2) by going with (1).

 

I am not sure I am convinced that (1) is a good idea unless the communities are completely independent. That is, logging into BBO1 would mean that you don't see your friends on BBO2.

 

My goals are to make sure that as bbo grows, we can manage the growth. This means a few things to me.

 

1. From a user perspective, the important one, I want to make sure that BBO will still remain usable when there are 1000 tables in the main bridge club and 10,000 users in the lobby. I have my doubts. I think we're already well past the point of comfort in terms of lobby size, table size, tourney size.

 

2. From a tech. perspective, the less important one, i want to make sure that I meet the ordinary goals of growth. The tech goals are: allow us to increase capacity by adding machines. Allow programs/machines to fail without taking down the whole system. Allow us to limit the total bandwidth consumed by any one machine in the network (we can add more machines easily but there are monthly bandwidth caps).

 

 

What I was thinking was that by spawning BBOs we'd find that the various BBOs could remain independent of each other. We could add new machines to add new BBOs. We could allow one BBO to fail without impacting others. Each BBO would be small enough to avoid using too much bandwidth).

 

This gains me all I want from a tech perspective.

 

Splitting BBO into fully independent services would be extremely easy. We have the resources to add as many machines as we need to add. As long as they stick to the bandwidth limit ( bbo is comfortably under it now), we wont have problems with CPU, network, HD, or anything else as long as we can just keep adding machines, each one independent of the others (I know this isnt the absolute truth, but it is close enough for now)

 

 

However....unless we can meet the primary objective, keeping the user happy, this won't work for us.

 

I was hoping that one way to do this would be to divide BBO up into mini-BBOs where people would naturally gravitate to one and see no reason to visit another, because their friends would tend to cluster within the same BBO.

 

Some ideas for BBOs that might fit this bill. Assume that all BBOs will only have this much in common: the interface will be roughly what it is today, you will have to select a BBO when you login.

 

 

ANARCHY: No rules. Do what you want. Nothing is forbidden, except complaining to us.

 

BIGBROTHER: All chat is recorded

 

PICKUP: what we have now, for random pickup games in the main club

 

 

OLDTIMES: What we have now

 

INVITATION: Can't get in unless invited in by a member. If you act out (rude,cheat) your sponsor will be censured.

 

 

VUGRAPH: just for vugraphs

 

ACBL: just for acbl games. Maybe also SKY/LAND/ITA etc

 

SUBSCRIPTION: Can't get in unless you pay an annual fee of some sort.

 

FREE_TOURNEYS: just for free tourneys

 

 

PAY_TOURNEYS: just for misc pay tourneys

 

POLAND/ITALY/FRANCE/etc: official language as you see it

 

PRIVATE CLUBS: just for private club games and tourneys

 

 

And so on.

 

 

To me the challenge is in coming up with a breakdown that makes sense, so that a person won't usually need to use the BBO software to track his friends on other servers. Maybe we can add a web page that lets you list/chat-with your friends. Maybe we can add a separate pop up window that lets you see which of your friends are online and where .

 

Maybe we'll find that some people and all their friends will play as a group on one of the servers and wont care about the others ?

 

 

to sum up: i am trying to determine whether fragmenting bbo into multiple servers is workable from your point of view, not necc. how to do it.

[cherdano]

Uday, I will say it again: no, I think this splitting up won't work.

 

You see, I would always try to choose the server where I am most likely to find most of my friends. If everybody does that, the only stable equilibrium is the point where everybody (well, maybe 90%) chooses the same server. It will be as successful as splitting people up into Main Bridge Club and other bridge club. (Maybe there will be a few special servers, like a total points server, that will attract a special small community.)

 

Unless, of course, you impose restrictions, which will make people unhappy.

 

There are some friends on BBO that I really like to play with, that I currently run into maybe once a month. If the split up succeeds, I would run into them once a year.

[luke warm]

i think this would happen: the star players would log on to the same bbo each time, unless they decide to segregate by nationality... if that's so, and if a lot of people think like i do (kib a lot, play with friends some, play with strangers rarely), then the bbo with the star players would always have more people than the other... probably a *lot* more people, which seems to defeat the purpose

 

i don't know if this is possible or not... log on to one "bbo"... from there, decide which bbo/priv club/vugraph, etc, to enter... if that can be done from one central location, then splitting by uday's suggestions seems pretty good... if it comes to it, we can always use msn or yahoo to find out which bbo to meet in (tho truthfully, i'd probably log on to whichever one gives me the best kibbing environment)

 

this next won't be popular... i know you want to keep bbo free, but i personally would not be against paying say $50, $75 a year... *especially* if that fee gave me unlimited (or mostly unlimited) free play in acbl and other pay tourneys, access to vugraph, etc... split the fee up, 50% (or whatever) to bbo, 25% to acbl, the other 25% to other pay tourneys... who knows, the acbl may even buy into something like that... say 2500 people paying $50 a year.. that's over $30,000 a year to the acbl

 

told you it wouldn't be popular :)

[DrTodd13]

I'd be interested in hearing Fred and Uday's thoughts on the other ideas suggested to improve scalability. Despite the predominantly negative reaction to the idea of mini-BBOs, it seems we're still trying to be sold on the idea. Nobody is asking for this as a useful BBO feature. If you want to satisfy user desires then you'll need cross-mini-BBO communication that will likely reduce or eliminate any scalability benefits. Please correct me if I'm wrong but it seems like you are highly biased to improvements that do not involve lots of new coding. As a programmer, I can understand this desire but from users who aren't programmers you won't get any sympathy. All they will know is they don't like the change. There seems to be some philosophical belief that this "all clients constantly in-synch with everything going on in BBO" is a desirable, useful property. From what I can tell, this philosophical position is

responsible for a large part of the scaling problems yet from what I've seen, most of the BBO user base doesn't care about this "constantly in synch" property. Indeed, most of them probably never think about and don't realize that while they are at a table playing they are constantly getting updates about what other people are online or what tables are open. I hate to beat a dead horse but in my view the two best things you can do for scalability is to drop the "constantly in synch" property and to distribute table hosting responsibilities to a cluster of local machines.

 

Todd

[fred]

I'd be interested in hearing Fred and Uday's thoughts on the other ideas suggested to improve scalability. Despite the predominantly negative reaction to the idea of mini-BBOs, it seems we're still trying to be sold on the idea. Nobody is asking for this as a useful BBO feature. If you want to satisfy user desires then you'll need cross-mini-BBO communication that will likely reduce or eliminate any scalability benefits. Please correct me if I'm wrong but it seems like you are highly biased to improvements that do not involve lots of new coding. As a programmer, I can understand this desire but from users who aren't programmers you won't get any sympathy. All they will know is they don't like the change. There seems to be some philosophical belief that this "all clients constantly in-synch with everything going on in BBO" is a desirable, useful property. From what I can tell, this philosophical position is

responsible for a large part of the scaling problems yet from what I've seen, most of the BBO user base doesn't care about this "constantly in synch" property. Indeed, most of them probably never think about and don't realize that while they are at a table playing they are constantly getting updates about what other people are online or what tables are open. I hate to beat a dead horse but in my view the two best things you can do for scalability is to drop the "constantly in synch" property and to distribute table hosting responsibilities to a cluster of local machines.

 

Todd

OKBridge used to (and maybe still does) rely of each table host's computer to be responsible for relying all table-related messages to all clients who are playing or kibitzing at that table. There are some problems with this approach:

 

- If the table host has a poor connection or slow PC then everyone at the table suffers

- There is a limit to the number of kibitzers who can join a table

- If the table host leaves, the table closes (you could get around this by writing additional code, but it would probably be a lot of difficult work)

 

So as far as I am concerned, any solution that involves depending on our clients to act as "table servers" is a non-starter.

 

I personally don't think "constantly in synch" is necessary, but it is certainly desirable in some areas. For example, if you are looking at a list of table in the hope of finding a place to play and if the list doesn't update itself automatically, you will frequently try to sit in an empty seat that has since been filled. Furthermore, having to click a "refresh" button constantly is a big pain. People with slow connections would also suffer every time they went to into the Main Bridge Club (or clicked the "Tables" button) because they would have to wait for the table list to download. The bigger BBO gets the bigger these problems get.

 

Some other comments about objections to "mini-BBOs":

 

- I am quite sure that those of you who think that the stars will all cluster on a particular mini-BBO are 100% wrong. Stars are just like everyone else. If they want to play in a pickup game (or with a student) it won't matter to them what BBO they are on, as long as they can find a table. Also, the group of stars is not a closely knit "brotherhood" or anything. Most of them have no idea who most of the other stars are. Keep in mind that probably less than 5% of stars are actually "famous players". The other 95% are just people who have represented their country in World Championships. Of course most of these are fine players, but few of them are good enough or famous enough to be a reason for other stars to care what BBO they choose to play on.

 

- My guess is that a massive % of our members won't care if they log in to BBO1 or BBO2, provided that both servers have serveral 1000 people logged in. I believe that most people log in to play or kibitz in pickup games. As long as there is a sufficicently large number of tables to choose from, nothing else matters. In fact, I suspect that many of our users will click the button that says "log me into the BBO server that will provide me with the faster connection" (or whatever). Surely have a quality connection is more important for almost anyone than which specific 500 tables appear in their Main Bridge Club list.

 

Don't take the fact that Uday and I have not commented much to mean that we are not paying attention. In fact, we are paying very close attention. I have personally read every post in both "scaling" threads several times. You may also want to know that before Uday's first post on this subject, we were both firmly convinced that mini-BBOs was the way to go. As a result of your posts we are now on the fence (and if anything we are tilting in the other direction).

 

So keep those posts coming - we are listening!

 

Fred Gitelman

Bridge Base Inc.

www.bridgebase.com

[QuiteACard]

Would it be that difficult to start to move some of the services (e.g. vugraph, "Explore Bridge", the private clubs) to separate physical servers without the players even knowing about it?

[fred]

Would it be that difficult to start to move some of the services (e.g. vugraph, "Explore Bridge", the private clubs) to separate physical servers without the players even knowing about it?

I would not be very difficult at all, but it would have a negibable impact on the problems we are going to have if our numbers continue to grow.

 

Fred Gitelman

Bridge Base Inc.

www.bridgebase.com

[hotShot]

Do we need a complete list of players loged on? I don't think so, if this list would contain only friends and yellows, this would do for a start.

So if someone logs on, just seen the information to those who marked him as friend, and give him a list of his friends that are online.

I would be willing to wait as long as it takes to get the compleat list when needed. (But i don't think i would need it, maybe i even know about the nick i'm looking for.)

But I would want to be able to klick on a tourney hosts/directors name to kontact them and i would like to be able to click on my partners name, to send him a message.

 

You have a point there with the table list, but you could separate those lists.

1) incompleat tables

2) tables with host names a-k

3) tables with host names l-z

4) tables with host names not starting with a letter

 

If you keep only 1) updated this should do. The others would require immediate updates only if i'm viewing this part (maybe a tabbed view) of the table list.

 

Updates of the tourney lists would be good enough when i look at it (at the table or by entering the tourney area).

 

 

Perhaps you know how many users have their Lobby chat switched off.

I remember when BBO had only a few hundred players online, it helped to ask for a player to join the table. My impression is, Lobby chat does not work for that purpose any more. (Probably everyone turned it off.)

Perhaps it is no longer needed for players, just for yellows to broadcast.

 

Your objection against using table hosts to distribute data is very valid.

 

But each table is a very indipendent unit to handle, tables could be load balanced to different servers easyly.

Using EJB one would think of a table as a "Session Bean".

 

A different approach would be possible if you use the apache webserver. You could use "mod_proxy" to transfer the workload to different PC's.

Any request send to e.g. bboserver/vugraph, would not be handled by the server but forwarded to another PC and it's answer to that request would be send back to the client. Your only limit would be the bandwidth to your ISP.

[QuiteACard]

Would it be that difficult to start to move some of the services (e.g. vugraph, "Explore Bridge", the private clubs) to separate physical servers without the players even knowing about it?

I would not be very difficult at all, but it would have a negibable impact on the problems we are going to have if our numbers continue to grow.

 

Fred Gitelman

Bridge Base Inc.

www.bridgebase.com

Is it fair to ask which resources are the limiting factor to BBO's growth as it exists today? I have reviewed the thread Scaling, Opinions/ideas solicited but there's not too much technical detail provided. In general is system growth constrained due to limitations regarding memory, processor, IO, network communication, or something else?

 

I understand this is an easy question to ask and almost impossible to answer definitively but I'm just trying to get a feel for the situation.

[Chamaco]

I do not know the details of the BBO code implementation and problems.

 

However, here is my view on scaling:

I like the idea of having on a separate server the activities external to the "Main LobbY".

 

So, basically, moving to separate servers:

- the Vugraph Broadcasts,

- all the Tourneys

- Team Matches

- Private Club Activities

- Partnership Bidding

- Access to Library

- Online BB Sample products.

 

BUT, my personal wish is that the Main Lobby stays as a single BBO.

 

As far as i can see, for example in this moment, out of a total of 5200 players logged in, only 1700 are in the main lobby, about 80s are in Clubs,

and about 1950 in the tourney/team matches.

The remaining are in the "explore bridge" section (Chat/Pship bidding, etc).

 

So, if we consider this 1/3 ratio of people in the main lobby vs total logged, even if the case 10000 of total BBO member logged in, theat would mean about 3300 players in the main lobby, which I guess (maybe wrongly?) that would be manageable ?

 

Just my 2 cents :-)

[cherdano]

Hi Fred,

 

I would like to add one more comment. In the potential setup described by you at the beginning of this thread, it seems to me that the only things I do not get from BBO2 when logged into BBO1 are the following:

• Full player list
  
• Full Main Bridge Club table list
  
• Lobby Chat
  

However, in my view all of them are unneeded anyway (of the first two, most people only need friends, star players, and open tables), and this seems to be almost a consensus according to my reading of the posts in the two threads. (Exception: Organizing team matches currently pretty much needs Lobby Chat, this would need a solution; an easier way to setup team matches would be really nice, anyway.)

 

If that is right, you could just disable all of them, and then there would be almost no user visible difference between the current setup, and a server-split into several servers.

 

Best,

Arend

[fred]

Would it be that difficult to start to move some of the services (e.g. vugraph, "Explore Bridge", the private clubs) to separate physical servers without the players even knowing about it?

I would not be very difficult at all, but it would have a negibable impact on the problems we are going to have if our numbers continue to grow.

 

Fred Gitelman

Bridge Base Inc.

www.bridgebase.com

Is it fair to ask which resources are the limiting factor to BBO's growth as it exists today? I have reviewed the thread Scaling, Opinions/ideas solicited but there's not too much technical detail provided. In general is system growth constrained due to limitations regarding memory, processor, IO, network communication, or something else?

 

I understand this is an easy question to ask and almost impossible to answer definitively but I'm just trying to get a feel for the situation.

The general nature of the problem is that, given the way BBO is now, every time any one of the following events happens we need to notify everyone on the system:

 

1) A person logs in

2) A person logs out

3) A person changes his profile

4) A person goes into or out of brb mode

5) A table is created

6) Table options are set/changed

7) A person joins or leaves a seat

8) A table is destroyed

9) A lobby chat message is sent out

10) A tournament is created

11) A tournemant's is edited or its status changes

12) No doubt several other things I have not thought of

 

With 5000 people logged in some of these things happen no more than once every second or 2, but some of these things happen dozens of times every second.

 

The total number of such messages that has to be sent out increases quadradically (sorry to the math people out there if this is the wrong word) as a function of the number of people logged in.

 

This is a problem because:

 

A) Computers, like people, can only do so much processing in a given amount of time. Eventually our poor server computer's CPU resources will become bogged down and be unable to handle the load. I actually find it remarkable that our server (which is just a PC) can handle as many simultaneous connections as it does now (3 cheers for Uday and Intel!).

 

B) The wire that leaves our server computer has a maximum capacity (in terms of bits/second). We will eventually exceed this capacity. Yes, we could get a thicker wire, but that would be expensive and would not solve the other problems.

 

C) The wire that enters your PC has a maximum capacity (in terms of bits/second). Dialup users will eventually be unable to swallow all of the data we are trying to shove down their throats.

 

D) Even if we get past A), B), and C), the client program still has to be able to process all the data and do all the other things it does (some of which, like drawing the screen and responding to mouse movements to say nothing of "serious operations" like clicking the DF button, can be computationally expensive).

 

Of these problems A) is our most serious concern. B) is our next most serious concern. C) could become an issue one day. As far as we can tell D) is unlikely to ever be a real problem (but we still want the client to be able to behave without any apparent "sluggishness").

 

Moving things like requests for Bridge Library articles to a different server will not make any real difference, because such requests are relatively infrequent and the server response to such requests is sent to only 1 person (as opposed to everyone who is online).

 

Hope that helps.

 

Fred Gitelman

Bridge Base Inc.

www.bridgebase.com

[DrTodd13]

OKBridge used to (and maybe still does) rely of each table host's computer to be responsible for relying all table-related messages to all clients who are playing or kibitzing at that table. There are some problems with this approach:

 

- If the table host has a poor connection or slow PC then everyone at the table suffers

So do something smarter. The table server doesn't have to be the table host. You could find the person at the table with the most reliable computer and then let them serve the table. I realize moving this function to clients would be hard work but you could just as easily have more computers onsite that handles serving tables. This seems completely distributable. Just have these servers request a new hand and then report the result back to the master server. All the table chat and play of each card could go through these servers.

 

- There is a limit  to the number of kibitzers who can join a table

Again, do something smarter. If you were to choose a client to host a table then pick the client that has broadband and if none of them have broadband then kick the table hosting responsibilities back to a BBO server.

 

- If the table host leaves, the table closes (you could get around this by writing additional code, but it would probably be a lot of difficult work)

It wouldn't be easy but I don't think it is something so difficult it couldn't be done. No rule says the table has to shut because the table host leaves. That is just a decision OKB made because it was easy.

 

So as far as I am concerned, any solution that involves depending on our clients to act as "table servers" is a non-starter.

All I'm saying is that it is an option that shouldn't be rejected out-of-hand. The problems are surmountable and in the end it is a much more scalable solution.

 

I personally don't think "constantly in synch" is necessary, but it is certainly desirable in some areas. For example, if you are looking at a list of table in the hope of finding a place to play and if the list doesn't update itself automatically, you will frequently try to sit in an empty seat that has since been filled.

I would find this preferable to what happens now, which is that I can't figure out where to play because the list is changing too frequently. There are some options here. One is a refresh button. The other is allow the user to specify an automatic refresh interval. Every X seconds the client automatically requests a refresh. Another possibility is that when the client is actively looking at tables the server automatically sends updates. At least this way, if you weren't looking at the table list then the client wouldn't need to be processing updates to table...neither would the server for that matter.

 

Furthermore, having to click a "refresh" button constantly is a big pain. People with slow connections would also suffer every time they went to into the Main Bridge Club (or clicked the "Tables" button) because they would have to wait for the table list to download. The bigger BBO gets the bigger these problems get.

When I was on OKB, I never minded the refresh button. The relevant table data should be so small anyway that even slow connections it doesn't matter. Inevitably, sending big messages is going to be more scalable than sending lots of little messages. The more you can collapse small messages into big messages the better off you'll be.

 

 

Surely have a quality connection is more important for almost anyone than which specific 500 tables appear in their Main Bridge Club list.

If I see one friend is on BBO1 and another friend is on BBO2, how do we get together and play? If it isn't transparent it is going to be annoying. I suspect this isn't the case for novice through adv tables but it can still be difficult to find a good expert pick-up game and splitting these people between more than one server will make the situation more difficult.

[hotShot]

1) A person logs in

2) A person logs out

3) A person changes his profile

4) A person goes into or out of brb mode

5) A table is created

6) Table options are set/changed

7) A person joins or leaves a seat

8) A table is destroyed

9) A lobby chat message is sent out

10) A tournament is created

11) A tournemant's is edited or its status changes

12) No doubt several other things I have not thought of

 

I don't think it would make much of a difference to me if you changed that to:

1) A friend logs in

2) A friend logs out

3) A friend/ someone playing/(kibbing) at my table changes his profile

4) A friend/ someone playing/(kibbing) at my table changes brb mode

5) See that a table is created only if i activate a table view

6) see 5

7) only if it's my table or i view the table list

8) see 5

9) Do you mean chat to Lobby (all), i don't need that i have lobby chat turned off

But it might make sence to create a team match chat room (smaller group)

10) Only need that information when i activate a tourney list view

11) see 10 or if i am registered or put myself on the partnership desk

12) Lets deal with them when they come up.

 

Worst thing that could happen is that it takes a few (milli)seconds, if i enter the table/tourney list, but I'm doing nothing else then.

 

Does your ISP support broadcasts on IP level? You create a "broadcast IP-number" at your ISP's router, that represents e.g. all IP-numbers of kibs at a vugraph match. If you send the data to this "broadcast IP-number" at your ISP's he will distribute the message to all clients in the IP-list defined before. Save bandwidth between you and your isp, but your isp will charge the full outgoing traffic of cause.