Reputation System

[nickf]

I am not an average BBO member.

I think most of you who post about software enhancements, ratings systems, BBO improvements should reflect upon this simple 7-word sentence and its implications.

 

It's my understanding that Bridge Base Inc is trying to deliver a product for the mass bridge market and as such needs to cater to their demands first and foremost.

 

nickf

sydney

[DrTodd13]

I guess Fred should be spending his time trying to figure out how to stop people from psyching then because that is what people seem to want most of all. Seriously though, sometimes you have to lead and not just react to what people think they want. Trying to play in the MBC with random players is a nightmare. That is my experience as a BBO member. I don't know if people would have suggestions on how to fix this but if Fred could come up with even a halfway solution it would be great. The solution of "get a circle of friends and only play with them" doesn't work a good portion of the time and I'd say that is true for many other people.

[Trumpace]

Personally I think it would just be a waste of the programming team's time implementing this.

 

Self-rating may not be perfect, but it is the people who abuse it that are the problem. This system does nothing to address the problem of people over-rating the ability of some and denigrating others. Additionally, it is very difficult to rate the performance of others with any certainty.

 

But most importantly I cannot see sufficient people caring enough to spend the extra time rating everyone.

 

I use a much simpler reputation system. If I believe that people are deliberately overstating their ability, they are marked as enemies. Seems to work fine.

 

Paul

Waste of time? I wrote my own bridge system that I tinker with occasionally and I implemented this feature. It took less than a day to implement. I'm even willing to give the core of the ratings code to Fred. So, I don't think this is a huge time sink.

Really? One day? How much time do you think it will take to integrate into BBO?

 

Did you think of:

 

1) What UI needs to be given, to show the users rating, means to vote etc and how much time it will take to code it?

 

2) Do you need to change the client server protocol? What changes need to be made? How much time will that take?

 

3) Other issues like: Say the user clicked on "Rate player" and waited 3 days before actually making a rating. Will it's weight be more/less, what?

 

Writing code to calculate new rating etc is pretty trivial, the harder part is to actually get the whole scenario working as a proper feature.

[kenrexford]

If the idea is to maximize the enjoyment of the average person in finding a pick-up partner, then reputation ratings sounds great. The idiots out there will rate the good players low and won't want to play with them. But, if you always use 4♣ as Gerber, your rating will be high and others like you will find you brilliant as well.

[DrTodd13]

Personally I think it would just be a waste of the programming team's time implementing this.

 

Self-rating may not be perfect, but it is the people who abuse it that are the problem. This system does nothing to address the problem of people over-rating the ability of some and denigrating others. Additionally, it is very difficult to rate the performance of others with any certainty.

 

But most importantly I cannot see sufficient people caring enough to spend the extra time rating everyone.

 

I use a much simpler reputation system. If I believe that people are deliberately overstating their ability, they are marked as enemies. Seems to work fine.

 

Paul

Waste of time? I wrote my own bridge system that I tinker with occasionally and I implemented this feature. It took less than a day to implement. I'm even willing to give the core of the ratings code to Fred. So, I don't think this is a huge time sink.

Really? One day? How much time do you think it will take to integrate into BBO?

 

Did you think of:

 

1) What UI needs to be given, to show the users rating, means to vote etc and how much time it will take to code it?

 

2) Do you need to change the client server protocol? What changes need to be made? How much time will that take?

 

3) Other issues like: Say the user clicked on "Rate player" and waited 3 days before actually making a rating. Will it's weight be more/less, what?

 

Writing code to calculate new rating etc is pretty trivial, the harder part is to actually get the whole scenario working as a proper feature.

You misunderstand me. I did _everything_ in one day...the core of the ratings calculation, the UI changes, display each person's rating in their profile, modifying the client/server protocol, etc. The idea though was that one day was a ballpark estimate and I engineered my system to make changes like this easy so it may be different for BBO depending on how they have organized their code and what GUI tools they are using. However, I wouldn't suspect this type of change to be particularly huge for any bridge system.

 

To answer your third question, the way I did my code was that ratings weight degrades over time (non-linearly). The length of time is calculated since you last played with that person and had nothing to do with when you actually rated them. The idea being that you aren't getting new information in the interim so your rating should be based on the last time you did get new information.

[Trumpace]

You misunderstand me. I did _everything_ in one day...the core of the ratings calculation, the UI changes, display each person's rating in their profile, modifying the client/server protocol, etc. The idea though was that one day was a ballpark estimate and I engineered my system to make changes like this easy so it may be different for BBO depending on how they have organized their code and what GUI tools they are using. However, I wouldn't suspect this type of change to be particularly huge for any bridge system.

 

To answer your third question, the way I did my code was that ratings weight degrades over time (non-linearly). The length of time is calculated since you last played with that person and had nothing to do with when you actually rated them. The idea being that you aren't getting new information in the interim so your rating should be based on the last time you did get new information.

Wow! What is your typing speed? :D

 

More questions:

 

4) What happens if many people try to rate the same person at the same time. Is there some locking mechanism?

 

5) How would the performance be for 5,000 users rating around the same time? 10,000? More?

 

If you have managed to implement a full fledged feature like a rating system, in a "ship quality" way, in just about one day, kudos to you.

 

Anyway, even if it is implemented in under 1 day, there would be the need for extensive testing to point out bugs, which could take days etc.

 

All I am saying is that it is not as simple as you make it sound.

[DrTodd13]

Typing long code sequences is a pretty small part of the average programmer's day.

 

Answer 4) In one sense, there is no such thing as "the same time." Only one ratings request can arrive at any instant of time. If the machine I was using had more than one processor then I might have used both of them which would have required a locking mechanism of some variety but as it was the application is single-threaded so no locking was required. Also, you really expect 5000 people to be rating the same specific individual at nearly the same time? There may be many ratings requests floating around at a time but most will be for different users.

 

Answer 5) The performance of the algorithm I developed is proportional to the number of ratings for the person. The cost per rating is only a few instructions plus a couple memory references. I can't give an estimate throughput until you tell me the average number of ratings per person. Back of the envelop computation for 100 ratings per person and 500 cycles per rating would be around 500,000 people's ratings computed per second. If ratings processing became too costly then just do it once per day or once per week. As it is, I don't see the problem with doing it inline.

 

I know it is not as simple as I make it sound but there was some intimation that doing this is months worth of effort. All I'm saying is that I don't think it necessarily has to be that long.

[Trumpace]

Typing long code sequences is a pretty small part of the average programmer's day. 

 

Answer 4)  In one sense, there is no such thing as "the same time."  Only one ratings request can arrive at any instant of time.  If the machine I was using had more than one processor then I might have used both of them which would have required a locking mechanism of some variety but as it was the application is single-threaded so no locking was required.  Also, you really expect 5000 people to be rating the same specific individual at nearly the same time?  There may be many ratings requests floating around at a time but most will be for different users.

 

Answer 5) The performance of the algorithm I developed is proportional to the number of ratings for the person.  The cost per rating is only a few instructions plus a couple memory references.  I can't give an estimate throughput until you tell me the average number of ratings per person.  Back of the envelop computation for 100 ratings per person and 500 cycles per rating would be around 500,000 people's ratings computed per second.  If ratings processing became too costly then just do it once per day or once per week.  As it is, I don't see the problem with doing it inline.

 

I know it is not as simple as I make it sound but there was some intimation that doing this is months worth of effort.  All I'm saying is that I don't think it necessarily has to be that long.

Now you have misunderstood me.

 

I mean the locking mechanism on the server. The Client APP can be single threaded. If there is no locking, the last rating that is updated would be one and you will might lose some rating information without any sort of locking.

 

For 5) I mean 5000 rating not just for a single person, but on the whole. I presume you will have some sort of a backend DB where you store the rating information? Read/Write access to that might have to be controlled (DB systems usually provide that automatically) using a locking scheme. DB accesses can be slow. To make if faster, you might have to cache stuff in the server (I presume you are doing it already), and in order to write to the common cache, you might need locking (again). Even to just view the (accurate) rating of some person, you would need to do a read.

 

Imagine a similar situation on BBO where the profiles are being changed all the time, and you could have multiple people changing the same profile.

 

Anyway, I think this side-discussion is just polluting the thread. So I will stop.

[DrTodd13]

The server and the client are single-threaded. DB systems are optimized for this sort of stuff. There are much more complicated servers than bridge servers. Don't worry about what effect this has on the server. Just worry about how much people like the results.

[foo]

If we want a player rating system, which I'm not sure we do, it should be as objective as possible. Something akin to the Chess's ELO.

 

Any system based on human opinion is too easily biased and too easily "gamed".

 

However, it should be noted that OKB tried to do this with the Lehman's... ...and that many feel it greatly damaged "the OKB experience". Even drove some players from OKB. Even some experts I know.

It became that unpleasant in their opinion.

 

 

At the least, we probably need to realize that someone's desirability as a player or partner on BBO is a vector with at least 3 independent variables:

 

a= how objectively strong a player are they

b= how well do people like them

c= how much / often do they participate on BBO

 

"a" and "c" can be easily calculated in an objective manner. "b" is a reputation system, with all the good and bad that comes with such.

 

Who I want to play with can greatly depend on my goals and my mood.

The Norman Kay's of the world (who rank high on all 3 metrics) are few and far between. (Even then I may not be able to stand the pace, glacial, of a player similar to NK)

...and I might not want to put up with a Lew Mathe-like player at some points no matter how skilled he is.