Mobile Benchmarks Need Work

It's ironic that the issues that deserve the most industry cooperation tend to be the most contentious ones. Few issues have been more contentious than performance benchmarking. Slowly, we're moving in the right direction, but more work is needed.

Benchmarks are widely used for evaluating anything electronic. To get the best scores, silicon and system vendors aggressively “optimize” for their target benchmarks. Sometimes these optimizations are more like manipulations. The technical press is littered with stories of unfair benchmarking practices, and what is reported is only a small portion of the common practices.

Benchmarks face other limitations, too. The rapid pace of innovation often makes it a challenge to accurately test all functions of a system in ways that reflect real user experiences across a wide variety of platforms. Image capture and editing, for example, may be handled by a variety of chips and APIs, frustrating efforts to make meaningful comparisons across Android, iOS, and Windows phones.

It's time for better standards in the mobile industry.

It's time for better standards in the mobile industry.

For the full story, see EBN sister site EE Times.

— Jim McGregor is the founder and a principal analyst at Tirias Research, and a contributor to EE Times.

23 comments on “Mobile Benchmarks Need Work

  1. Taimoor Zubar
    July 30, 2014

    @Jim: What do you think the source of these benchmarks should be? Should they be driven by the companies that manufacture equipment? Or should they be driven by consumer groups? Both of these approaches have their own merits.

  2. ITempire
    July 31, 2014

    Jim, the question is who takes on the responsibility of publishing benchmarking material. Will someone do all the hardwork for earning just advertising money on website ? Do the users of that data agree that the publisher is unbiased ? 

  3. ITempire
    July 31, 2014

    Taimoor, just an observation, I think these are run by analysts who are both users and manufacturers. The mix does the best and the editor of such websites or the publisher of material should have the responsibility to ensure that the analysis and facts go through unbiased.

  4. ahdand
    July 31, 2014

    @Waqas: Plus the software or the tools they use to get the raw data analyzed and turn into golden ideas. 

  5. ahdand
    July 31, 2014

    @Waqas: I think it depends on the situation

  6. ahdand
    July 31, 2014

    @Taimoor: Well is there any strict policy written for this ? 

  7. ITempire
    July 31, 2014

    Nimantha, as long as the users get the facts, the analysts are doing great. The objective is clear. It is just a question of how it will be delivered.

  8. ITempire
    July 31, 2014

    In your opinion, who is better suited to provide the information that the users need? Is it the users themselves or is it the manufacturers?

  9. Ashu001
    July 31, 2014


    How do you expect Users to agree and Decide on the right Standards going ahead here?

    Most Users don't really have the Technical Training and Knowledge Base required for this action.

    They don't even have the patience to sift through reams of Data and Research to arrive at an effective decision.

    It has to be the right combination of Regulators,Government Agencies ,Independent Researchers & Manufacturers.

    How you get everyone on the same page today is very much a challenge.

  10. ITempire
    July 31, 2014

    Tech4people, I think users' involvement is also very necessary because they will be able to communicate their requirements and the difficulties they face in interpreting or putting into use the benchmarking data.

  11. itguyphil
    July 31, 2014


    Agreed. The best way to set goals is to get feedback from the people impacted first-hand. Instead of using projections & other metrics that do not implicate actual usage patterns.

  12. ITempire
    July 31, 2014

    pocharles, who do you think comprise the teams that publish benchmarking data ? Do manufacturers generally trust such websites, journals, magazines, etc. ?

  13. ahdand
    July 31, 2014

    @waqas: They should otherwise it will never happen. Someone has to take the initiative

  14. ahdand
    July 31, 2014

    @pocharles: Indeed, you do need your customer feedback plus you do need to keep in touch with them. Update them and look after them. Then only they will come back to you. 

  15. ahdand
    July 31, 2014

    @waqas: Yes and they can do it with the help of the tools easily. 

    August 3, 2014

    I am not a great Iphone fan but I saw some aspects of the Iphone's perfromance to be far superior than its Android competiitor even though the latter fared better on most accepted benchmarks.  Maybe it is application dependent too?

  17. ITempire
    August 17, 2014

    Nimantha, the reliability of the data is the key no matter where it comes from.

  18. itguyphil
    August 19, 2014


    For the large, well known publications, yes I do think manufacturers trust. The not so well-known, probably not.

  19. ITempire
    August 21, 2014

    Pocharles, hmm. It is good if industry relies on such publications and that adds additional responsibility for publications to remain unbiased.

  20. itguyphil
    August 27, 2014


    I agree. Whether or not it actually gets put into place is another story.

  21. Ashu001
    August 31, 2014


    Fair enough.

    Unfortunately(as I am sure you would also agree wholeheartedly) is that Users routinely get shafted in the name of Cost-cutting or Contingency Planning or whatever else such Nonsense the Corporate Types come up with routinely.

    Sad,Sad story.

  22. ITempire
    September 21, 2014

    tech4people, yes it is true. I think if the organization's leadership has a clear idea of objectives and the strategy required to achieve it, they may agree to keep users onboard while taking any decision regarding limiting options for use of data. Sometimes a cost-cutting may seem good for the financials but it will be bad for processes and systems required to achieve sustainable growth.

  23. Ashu001
    September 25, 2014


    Absolutely True!

    Cost-Cutting cannot and will not ever be the Be all and end all solution for All of A Companies problems.

    You have to think smart rather than resorting to the Default mode of Operation today.

    That's the right way to go about this issue.

    Innovation as we speak.


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.