Inconsistent & Misleading Reviews: Can We Fix Them?

The blogosphere of Android has something for everyone – sites that focus on geek lifestyles, others that center on hardware, and then there’s websites like XDA that put a strong emphasis on the software and what it can do. Just like with Android itself, the diverse options allow anyone to find a favorite site or even groups of journalists. Youtube also has many important Android channels full of interesting analysis and videos. When compared to the iPhone scene, Android online journalism is much more interesting to say the least.

Yet there is something that we can’t help but notice at the XDA office, and that is that subjectivity is increasingly overtaking the scene. By this we mean that many stories, coverages and especially reviews tend to have a more “editorial” feel than what Android power users want. Hardcore Android fans are obviously not the only demographic that the internet takes into account, but in this quest for mass appeal we can’t help but feel that the quality of features and analysis, overall, drops dramatically. This is something that is most noticeable during phone releases, as hundreds of sites rush to put out a review that will net them recognition, awareness and of course hits. Chances are that if you agree so far, you will know the culprits. But reviews are where the inconsistencies show. Let me explain:

There are reviews that specialize on the technical details of new hardware: how its processor performs, how accurate the screen is, how long a battery lasts in comparable terms. These reviews take a lot of testing (and benchmarks) and thus, time. There are sites that excel at these kind of reviews, and we can not be more grateful for their existence. Then there are holistic reviews that do not require the same amount of time nor testing, and focus more on the user experience and what the reviewer thinks about the phone. This format in itself is not bad – in fact, it is needed, but the problem comes when the holistic findings contradict rigorous measurement, testing, other holistic reviews and sometimes even logic itself. And even worse, sometimes said reviews read like a salesman script.

At our XDA chatroom we tend to discuss the latest reviews for the latest phones, and as more come in we notice that sometimes they are too subjective. Things like battery life, for example, depend on what the user does with the phone, and this is something we see contradicted all the time. Sometimes a phone has mediocre battery life and it gets passed off as great, and sometimes it has great battery life and it gets passed off as mediocre. Technical reviews do a fair job at estimating battery performance by testing it with a common methodology on every device, which produces results that can at the very least be compared to others for a general idea. If a review does not break down the recorded usage pattern properly to the audience, it is very hard for readers or watchers to form an objective idea of how it would perform in comparison to other phones.

Apart from contradictions between sites, we also see statements that directly contradict physical reality. For example, Samsung devices have had a history of saturated screens, but the saturation has dramatically decreased with each of the past few panel generations. The Note 4, in particular, features a rather color accurate screen, as does the S6. That didn’t stop many reviewers from saying that the screens are still saturated. When it comes to performance, holistic reviews have small blurbs that do not give much depth as to how it actually performs and they sometimes mitigate the issues that faulty processors or bloated software might bring.

Which brings us to a bigger problem: a small number of reviews seem afraid to point out faults in a device, or to criticize certain aspects of it. We see a lot of ambiguous or vague descriptions for some device cons, for example.  A review should not hide nor mitigate issues nor frame them as insignificant to users that might very well care about such problems. Not every review has to be positive, and if a device severely underperforms in key aspects it should always be noted. All of these factors and more cause many reviews to be uninformative and contradictory, and in most cases misleading as well. To reiterate, we’ve seen this more times than we can count, and we’ve seen many users here, on reddit, youtube and those very sites realise that as well. Some devices like the Nexus 9 saw levels of inconsistency that suggest that there is something seriously wrong here, and devices in 2015 still see these kind of reviews (I’m looking at you, G4).

There is an Android for you, and finding it could be easier. We just need to know how they compare!

Now, many of these faults are due to human error. We all make mistakes, and no journalist is perfect nor knows it all. Holistic reviews simply do not yield accurate results either, and everyone uses Android differently. We do not put our site above others in these regards, but I know for a fact that our team works very hard to inform users truthfully, because that is what XDA’s mantra is all about. We are known to speak out against the companies that transgress upon consumers’ rights or respect. Ultimately, we try to be as fair and honest as possible – I hope that you, our readers, see us that way. If every site did technical reviews and found similar or the same data results, then reviews would also get pretty boring. After all, we also read and watch Android content for the personalities behind it, some of which we grow very fond of.

We do believe, however, that something must change – because otherwise, many consumers, users and Android fans will get misled or misinformed at some point or another. How can we do that? We have a few ideas in mind that we want to expand upon to hopefully agree on a standard with the community at large. We will let you know about them soon. For now, we just have to wait. Not all sites and blogs commit all faults nor do they commit them at the same degree. We are most likely not exempt from some of this, but we do try to focus on being informative, technical and thorough. Until reviews become more objective and informative, though, I personally will stick to my handful of in-depth reviewers. We want to hear what you have to say about this, as the power users behind the news portal discuss this almost daily. So we ask you:

What do you think of inconsistent reviews? What would suggest reviewers did to fix this? Discuss it with us in the comments!

The post Inconsistent & Misleading Reviews: Can We Fix Them? appeared first on xda-developers.

from xda-developers http://ift.tt/1E27geV
via IFTTT

from XDA http://ift.tt/1ARR0gv
via IFTTT