Discussion: Rating system for TPFC

[Andrew Lee]

@Fluffy *bittorrent of abuse* ... Twisted Evil sersiously, your comment about a rating system struck a chord with me because I've long held the view that the "comments" are pretty much pointless. People use them to bitch and complain about how they can't get this to work, or they can't find this feature, or program x is better at doing action y etc. etc.

However, I would also caution against a simple 5 star rating system too because that doesn't really tell you much either, for example suppose that an application is super efficient at what it does but it has a user interface from the bowels of hell, how do you rate the app? Anyway, I digress, perhaps this is one for another topic. I don't want to distract from El Salvador's excellent work.

I am all for a rating system, but my tiny 8-bit brain can't think of a rating system that:

1) can't be gamed
2) provides you with useful information about the app, other than the number of stars

At least with comments, I try to delete stuff that I feel are not related to the app in question, so the remaining ones should at least convey _some_ information.

Would love to hear your comments.

[Darkbee]

Well I'm sort of going to contradict myself a little bit and say that the most important thing is to keep it fairly simple (however, not SO simple as to use a single 5 star rating system). I probably don't need to go into the dangers of making the rating system so horrendously complicated.

My instinctual reaction is to 'borrow' from other software directories on the web, that allow for ratings across multiple categories, and then a general overall rating. Not sure whether the overall rating should be a calculated average of the other category ratings or user defined. However, with a user defined overall rating you have the possibility of giving an overall false representation of the average ratings (i.e. rate all categories 1 star but give overall rating 5 stars which seems nonsense). On the other hand, having a calculated average may be more susceptible to "gaming" (which I assumed to mean 'rigging', 'fixing' or otherwise mis-representing). This is because users may realize that if they give true category ratings then a given app has no chance of getting 5 stars, so they rate all categories 5 stars (or of course the reverse, rating all categories poorly to get a poorer overall rating). Either way people can *work/abuse the system. * delete as applicable

If you choose to have a series of categories, the question then arises of what should these categories be. This is a tricky one to be honest, but is crucial in getting right so that people feel that they can accurately sum up the overall value of a given app. Plus, naming the categories should leave little room for multiple interpretations. Probably one of the ways of generating categories is to look at comments and see how people tend to compare apps. Common comments often deal with the look and feel of an app e.g. app x looks nicer/is prettier than app y, OR app x seems easier to use/is more intuitive. Other common comments seem to deal with performance e.g. app x seems to perform faster/better than app y, app x seems more resource hungry/uses more memory than app y etc. So two categories could be User Interface and Performance (although even 'performance' is quite open for interpretation). However, at this stage of the game I'm more about putting ideas forward rather than propose a full blown rating system myself.

I think comments have their place too if used in conjunction with the ratings, to perhaps explain why certains ratings were given e.g. I rated the user interface of this app as 2 stars because it is not very intuitive and seems extremely unresponsive.

The main problem with ANY rating system (no matter how simple or complicated) is that it is purely subjective and one man's trash is another man's treasure so to speak.

As a couple of extra tack on thoughts, naturally any rating system would have an explanation of it on the main website, so users could be sure that they were using it correctly. Another thing is that we would have to make certain assumptions about the user, for any ratings that they may give. To explain a bit further, we assume that they know what the hell they are doing, users who can't even install the software correctly should not be rating it; we assume that they have it working correctly. We also assume that, and this kind of ties in with the last comment, the user is running the application on hardware/software that meets the required specifications. For example, a user running an application that requires a computer with a minimum of 64MB RAM on a 32MB machine should not really consider giving comments (particularly if the comments are negative claiming that the application is unresponsive or doesn't work correctly). Naturally, we have no way of verifying any of this for any users and any rating system has an inherent amount of trust built into it but it's just something worth keeping in mind.

I pass the torch on to someone else with the *promise/threat of returing with more comments as I have them. * delete as applicable

[Fluffy]

Honestly, I was thinking of "picks." Basically, since I adore the program DM2 and use it all the time, it could be tagged with "Fluffy's Pick." An application like FileZilla could be branded "Andrew's Pick" and whatever the hell Darkbee likes could be branded with "Darkbee's Pick." Though this might seem a bit elitist, I think that all 3 of us are software connoisseurs and we are pretty well-grounded in software functionality, ease of use and overall presentation. I'm not saying that other users can't be added, but this would be one way to help avoid abuse of a ratings system by giving certain level-headed and experienced users the ability to nominate outstanding programs. The idea is that some of the more solid and spectacular apps like FileZilla would probably have multiple nominations and stand out among the crowd to help people recognize them.

One thing to clarify, though, is that users that ask for their own picks section would be denied as it's an honor given by Andrew, Darkbee and I. Once again, it probably seems a bit unfair and "playing favorites," but I would rather have such a system in place than an easily abusable rating system.

Plus it'd be nice to get a bit of a staff page going on to give novice users a place to start. A huge database like TPFC can undoubtedly be a bit intimidating to anyone - I mean, for a novice user who wants to start moving to a portable setup it's probably pretty overwhelming to just see hundreds upon hundreds of entries and wonder "well, where should I start?" In this scenario, they could just check out one of our profiles and see what we recommend.

[Darkbee]

It's an interesting idea but one that I'd prefer to see alongside a user-wide rating system because, as you said yourself, there are literally hundreds of applications in the database and I use (or have used) only a very small percentage of them. Furthermore, already in the very small list of apps that you mentioned (i.e. one app, DM2), I happen to agree with you that it is a very useful app, so we might end up with a lot of over-lap.

For me (and it could be different for others) the purpose of a rating system is to help me identify which applications I should try first, when trying to find a 'new' app for a particular need I have. Now, to a certain extent I can, and do, use the comments to do this but in a very flippant way. The number of comments there are is a very rough way to estimate the popularity of a given app. Granted it isn't fool-proof and that is why I dispise comments like "app sucks", "Can somebody tell me how to turn on my computer?". Generally speaking, the app with more comments obviously attracts more attention and so is probably worth checking out (unless of course all the comments say "this app is the worse piece of software ever written and should be avoided at all costs" or words to that effect). However, unfortunately a lot of the comments I've seen are not of the helpful variety and I usually find it more helpful to simply ask people on the forum for their opinions. It'd be nice if I could get a very quick but reasonably accurate assessment of the overall popularity and usefulness of an app with a few simple clicks, without relying on, or waiting for people to respond.

This makes me think of a different type of rating system but perhaps as equally as valid/useful. Instead of arbitrary star ratings, have simple yes/no or multiple choice questions for each app like "Do you use this app on a regular basis?" (100 people say yes, 10 people say no), or "How often do you use this app?" (daily, weekly, other frequencies, never etc), "Would you recommend this app to others?" (yes/no). And if you want to get a little bit fancier then you could have like Amazon does; "users of this app also recommend... " etc.

Again, just throwing some ideas in the pot. The more I think about it, the more it seems that just a very few simple questions or rating scales could go a long way to help identify the best of the best.

...and another quick thought, I'm assuming that once a user (forum user?) has rated an app they either would not be able to rate it again, or have the option of modifying their existing rating/s. That's probably the most obvious form of abuse right there if I go rate DM2 500 times as the best app ever. It's a pretty good app but not that good!

[usdcs]

This topic is somewhat analagous to whether one pays attention to movie reviews or the box office. It boils down to whether you usually agree or disagree with the particular reviewer. And/or, do you usually follow the crowd?

I'd like to see a "Top Picks" Forum as opposed to a ratings system. A ratings system could / would start a bunch of unnecessary wrangling over the ratings themselves. Who needs that?

An Andrew's "Top Pick" would be just that. It's Andrew's subjective view. I could then decide whether I agreed or disagreed with him. I couldn't argue with his assessment. Andrew, Darkbee, and Fluffy could have their own Sticky threads at the top. The rest of us could post our top picks, if we like, in our own thread.

Another Sticky thread could contain the suggested way to "rate" or to document our Top Picks. But, keying in on Darkbees comment, how many multiple choice questions should we have? Some people may be interested whether an application does the job, or not? Others may be interested in multiple language support, or Memory footprint, or stealth vs. portable, etc. How many questions would be enough? How many would be too many?

Recently, the PowerTools Forum on Yahoo ran a thread called "The indispensable twelve" where people listed their 12 (and sometime more) favorite applications. I found it enjoyable to read people's comments and, particularly why they chose a particular application.

So, I'd like to see the "Favorite App" thread expanded into is own Forum. Would there be overlap? Certainly. But, it would give everyone the opportunity to be a reviewer and wouldn't carry the overhead of establishing and maintaining a "system."

[Andrew Lee]

I like the "picks" approach, though I tend to agree that it is more useful for everyone to participate, rather than restrict it to a chosen few.

Following the thoughts of usdcs, suppose we allow everyone to pick or recommend a maximum of 10 or 20 apps. For every app that is picked, the user has to type in a comment to describe why he has picked that application. By aggregating everyone's picks, it will be possible to find out which apps are more popular. In addition, by effectively giving the user a limited number of votes, he has to think hard about which apps he wants to allocate his votes to. The compulsory comment also increases the effort required, hence making sure that only serious players will cast their votes.

Assuming we can integrate the voting system with the web forum user database, the system is still susceptible to gaming. For example, if I want to promote a particular app X, I can register 100 forum accounts, then pick X for all the accounts. The problem is, this kind of behaviour is very difficult to detect and correct.

Maybe, by requiring a minimum number of words for the comment (hence eliminating useless ones like "This rocks" or "Good!"), and making these comments easily viewable as a list for any particular app, it will increase the barrier to gaming the system, since the abuser will have to think of 100 different comments for app X which do not stand out in a list view.

[Darkbee]

Maybe, by requiring a minimum number of words for the comment (hence eliminating useless ones like "This rocks" or "Good!"), and making these comments easily viewable as a list for any particular app, it will increase the barrier to gaming the system, since the abuser will have to think of 100 different comments for app X which do not stand out in a list view.

I don't see anything wrong with this approach but you're never gonna take the junk out of junk mail

Now we just have to decide on how many apps everyone should be allowed to pick. I'll have to go and review my "picks" to see what kind of number I come up with. You don't want to make the number too high otherwise it becomes a burden.

As before, I'm assuming we'll be able to edit picks since they are likely to change even if it's just a case of replacing one app with another, better one.

[Fluffy]

For me, there's a lot of games I'd like to nominate so I'd have to just cover the basics otherwise.

Letting everyone participate is a nice idea in theory, but it opens up the door for trolls and flamers (and also folks that really have no idea what they're talking about but want to "promote" or "demote" an application accordingly). It's a hard decision either way.

[technicolordreamcoat]

would it be too difficult to allow a rating system with several filters:

filter one: picks of chosen people like admins and moderators
filter two: rating system available to everyone

kind of the rating system on www.download.com

this way, the staff as well as the general public can voice their opinion
(just thought about another filter: by downloads)

i think the more filters you have, the easier it would be to build an objective
opinion

$0.02

[AlephX]

Hi!

I see that this topic (users´picks) is quite complex and dangerous (see what Fluffy wrote about trolls...)

I think that drumcolorbeat´s idea of a double pick sistem would be good, also if limited to 20-30 picks per user (see Andrew)...

Maybe the user limitation could be 2 picks per category (so many categories in the Collection!) thus the picks would be like an internal votation (like in Pricelessware sites)
Exception: what Fluffy says about the fact that many games would be already "picked"... so the "editors" (Andrew, Fluffy and Andrew) can pick what they want, while users have limitations.
Or the "games section" can have a different status...


ANother point concerning portability.
Some users need an absolute portability, others don´t care of "purism"... it would be good to tell the user what kind of portable program is he using.

for example (modifications really welcame):

4= absolutely portable
3=portable (not important setting in the registry)
2=portable (like 3, but with little nags)
1=portable if necessary (only certain apps which write settings, have some nags, but can be useful and work as portable)
0=rejected (not in Database)

portability = 1 implies that the app cannot have a pick as portable app, even if a very good one.

This portable-index would be (in my opinion) good for this Site which always had a serious policy of program selection

[Andrew Lee]

As before, I'm assuming we'll be able to edit picks since they are likely to change even if it's just a case of replacing one app with another, better one.

That's the intention.

kind of the rating system on www.download.com

I assume you are talking about "Editor's pick" and "User's favorite". Like Darkbee mentioned, I think the three of us (myself, Fluffy and Darkbee) will have very limited coverage. I think a scheme that takes everyone's opinion into account is best, rather than split into two sections. Alternatively, we can assign slightly higher weightage to certain users' picks.

just thought about another filter: by downloads

Actually this is already implemented in a way. Whenever someone clicks on "Launch" or "Download", a score is kept (duplicate IP addresses filtered). This is used to generate the "Popular titles" box that you see on the right-hand side of the page.

Maybe the user limitation could be 2 picks per category

I think it is much easier to recommend what I already use, without regards to the categories. It will be too troublesome to keep track of categories at the same time with recommendations!

ANother point concerning portability.
Some users need an absolute portability, others don´t care of "purism"... it would be good to tell the user what kind of portable program is he using.

I think this point has previously been suggested and discussed before. I am still of the opinion that certain types of apps (editors, viewers etc.) that write to the registry should be rejected instead of being added to the database with some kind of low "portability score". Reason is because otherwise 99% of existing PC freeware will be added to the database!

For those apps that do write to the registry but which we think is acceptable due to the nature of the app, we do try to note it down under the "Write settings to" field so the "purists" (I like to refer to them as the "stealth guys") can avoid them.

Besides "stealth", there is yet a more extreme school of portability purists that demand that portable apps must be able to run with "guest" privileges! I have once considered putting this as a field in the database, but didn't do it because it will be too much work for testing.

[Darkbee]

I don't think the "two picks per category" is necessary since how many notepads can one person use? if you're talking about broader categories like say "Audio", then particular individuals may work extensively with audio files but rarely have anything to do with images. Forcing that individual to pick their favorite icon editing program wouldn't really give any true value to other users, since it wouldn't be their area of expertise.

[dasfx]

I think some type of rating system would be nice. That said, I know that my use for the apps on here (as I think evident from my submissions) is more towards the Sys admin side (not that there is anything wrong with games ). I did like the idea of having the best of both worlds, that is, both user ratings and "editor" ratings, as technicolordreamcoat said.
I think another interesting idea would be something where users could submit what is one their drive (some site has a "my rig" feature where users can post specs of their systems). I know that I have a few apps on my drive are ones that I discovered in a "what's on your drive" type post on portableapps.com.

Dasfx

[orchus]

Pick lists are a great idea
There could be a list of the most full featured tools to Browse the Web - Download Files - Edit HTML & Descriptor Files and check them for portability *Grin*
Or you could have a list of the fastest (lightest CPU & RAM utilizing) programs to view & edit txt, rtf, htm & PDF (or images or video or audio)
Or the smallest programs (in Mb's on the USB) to do a given task
Or the "most-fun" portable games [very subjective - but often there's agreement - lots of people thought Doom & WarCraft were fun]
The idea is we can all have different ideas of what are good goals, for our portable programs to achieve
- the same people may be interested in posting Picks that seem contrary (like 1 member could want the smallest set of tools for some days and the most features on others - may depend on which computers they'll have available at the lab)
All their input could be useful


But how can abuse be minimized, without squelching creativity?


What if people's favorite "picks" were regulated Amazon List Style?

http://www.amazon.ca/Need-Your-Own-Pers ... XBTTFKMY1/
(Link to Amazon 'Need your own Personal Chef?' Book Suggestons)
(Ours of course should include much more explanation of what makes our picks fit together so great)
(Picks must refer to programs in the database, although coments can refer to those not in the database for comparisons)
(to prevent automated entry - users must enter text from an image like most fileshare sites & free e-mail registrations require)
{Please note: I hate those image entries - but they do tend to deter abuse}

To discourage abuse, each pick has a "Report Abuse" Link under it that can flag a pick-list for probationary review. (If it is determined to be spam the material is removed & poster can lose membership)
Posts with more than 10 identical words (in the same verbatim order as a pick-list posted in the previous week) are automatically flagged as possible abuse
Also No one can post an "officially-approved" "pick" list till they've been a member in good standing for a certain period of time (like a month) with at least 3 serious posts.
Unofficial Pick-lists are welcome from new members in a seperate section and can be considered as part of 3 serious posts.
Also readers can click to indicate if they found a pick-list useful (only official members can vote).
(..of course this "useful" rating is subjective & could also be abused)
But Amazon seems to be working so far


All the posts in the forums are easier to abuse, but most posts seem to be useful.
Most disccussion groups seem easy to abuse, but most contain useful information.
Maybe it depends on the number of abusers in the user base.

[AlephX]

Hi!

I think this point has previously been suggested and discussed before. I am still of the opinion that certain types of apps (editors, viewers etc.) that write to the registry should be rejected instead of being added to the database with some kind of low "portability score". Reason is because otherwise 99% of existing PC freeware will be added to the database!

For those apps that do write to the registry but which we think is acceptable due to the nature of the app, we do try to note it down under the "Write settings to" field so the "purists" (I like to refer to them as the "stealth guys") can avoid them.

Besides "stealth", there is yet a more extreme school of portability purists that demand that portable apps must be able to run with "guest" privileges! I have once considered putting this as a field in the database, but didn't do it because it will be too much work for testing.

Yes, I see all your points. I understand that this new field would be trouble-maker (for you as administrator and programmer of TPFC). But it would be useful. I think of the site http://www.gamehippo.com/ where they have (maybe too many) little icons to define certain parameters.

By the way they have a double rating system and the editor´s choice. Of course I don´t promote this way (see this thread ), but it can be interesting to see.

Anyway, if it creates problems (a portability index), better not to use. I thoght only to be aware of different portability needs (stealth people or "guest privilege" aware or "no-care-at-all" folk). And 99% apps would be not possible because the policy of this site (which I like)