It's ironic that the issues that deserve the most industry cooperation tend to be the most contentious ones. Few issues have been more contentious than performance benchmarking. Slowly, we're moving in the right direction, but more work is needed.
Benchmarks are widely used for evaluating anything electronic. To get the best scores, silicon and system vendors aggressively “optimize” for their target benchmarks. Sometimes these optimizations are more like manipulations. The technical press is littered with stories of unfair benchmarking practices, and what is reported is only a small portion of the common practices.
Benchmarks face other limitations, too. The rapid pace of innovation often makes it a challenge to accurately test all functions of a system in ways that reflect real user experiences across a wide variety of platforms. Image capture and editing, for example, may be handled by a variety of chips and APIs, frustrating efforts to make meaningful comparisons across Android, iOS, and Windows phones.
For the full story, see EBN sister site EE Times.
— Jim McGregor is the founder and a principal analyst at Tirias Research, and a contributor to EE Times.
@Jim: What do you think the source of these benchmarks should be? Should they be driven by the companies that manufacture equipment? Or should they be driven by consumer groups? Both of these approaches have their own merits.
Jim, the question is who takes on the responsibility of publishing benchmarking material. Will someone do all the hardwork for earning just advertising money on website ? Do the users of that data agree that the publisher is unbiased ?
Taimoor, just an observation, I think these are run by analysts who are both users and manufacturers. The mix does the best and the editor of such websites or the publisher of material should have the responsibility to ensure that the analysis and facts go through unbiased.
@Waqas: Plus the software or the tools they use to get the raw data analyzed and turn into golden ideas.
@Waqas: I think it depends on the situation
@Taimoor: Well is there any strict policy written for this ?
Nimantha, as long as the users get the facts, the analysts are doing great. The objective is clear. It is just a question of how it will be delivered.
In your opinion, who is better suited to provide the information that the users need? Is it the users themselves or is it the manufacturers?
Waqas,
How do you expect Users to agree and Decide on the right Standards going ahead here?
Most Users don't really have the Technical Training and Knowledge Base required for this action.
They don't even have the patience to sift through reams of Data and Research to arrive at an effective decision.
It has to be the right combination of Regulators,Government Agencies ,Independent Researchers & Manufacturers.
How you get everyone on the same page today is very much a challenge.
Tech4people, I think users' involvement is also very necessary because they will be able to communicate their requirements and the difficulties they face in interpreting or putting into use the benchmarking data.
@WaqasAltaf,
Agreed. The best way to set goals is to get feedback from the people impacted first-hand. Instead of using projections & other metrics that do not implicate actual usage patterns.
pocharles, who do you think comprise the teams that publish benchmarking data ? Do manufacturers generally trust such websites, journals, magazines, etc. ?
@waqas: They should otherwise it will never happen. Someone has to take the initiative
@pocharles: Indeed, you do need your customer feedback plus you do need to keep in touch with them. Update them and look after them. Then only they will come back to you.
@waqas: Yes and they can do it with the help of the tools easily.
I am not a great Iphone fan but I saw some aspects of the Iphone's perfromance to be far superior than its Android competiitor even though the latter fared better on most accepted benchmarks. Maybe it is application dependent too?
Nimantha, the reliability of the data is the key no matter where it comes from.
@WaqasAltaf,
For the large, well known publications, yes I do think manufacturers trust. The not so well-known, probably not.
Pocharles, hmm. It is good if industry relies on such publications and that adds additional responsibility for publications to remain unbiased.
@WaqasAltaf,
I agree. Whether or not it actually gets put into place is another story.
Waqas,
Fair enough.
Unfortunately(as I am sure you would also agree wholeheartedly) is that Users routinely get shafted in the name of Cost-cutting or Contingency Planning or whatever else such Nonsense the Corporate Types come up with routinely.
Sad,Sad story.
tech4people, yes it is true. I think if the organization's leadership has a clear idea of objectives and the strategy required to achieve it, they may agree to keep users onboard while taking any decision regarding limiting options for use of data. Sometimes a cost-cutting may seem good for the financials but it will be bad for processes and systems required to achieve sustainable growth.
Waqas,
Absolutely True!
Cost-Cutting cannot and will not ever be the Be all and end all solution for All of A Companies problems.
You have to think smart rather than resorting to the Default mode of Operation today.
That's the right way to go about this issue.
Innovation as we speak.