Please Note: This article is written for users of the following Microsoft Excel versions: 2007, 2010, 2013, 2016, 2019, and Excel in Office 365. If you are using an earlier version (Excel 2003 or earlier), this tip may not work for you. For a version of this tip written specifically for earlier versions of Excel, click here: Avoiding Rounding Errors in Formula Results.

Avoiding Rounding Errors in Formula Results

by Allen Wyatt
(last updated December 29, 2018)

3

Nick notes that the formula "=0.28*100-INT(0.28*100)=0" returns False even though it is obviously true. He believes the issue has to do with rounding and how the computer uses binary arithmetic, etc. Nick is using the formula as a part of a larger IF statement, and he assumes there are other rounding errors in Excel that can reach out and bite him. He wonders if there is a simple way to avoid formula pitfalls such as this.

Before looking at simple ways to avoid this type of problem, it is important to understand why the problem exists. As Nick notes, it really has to do with binary arithmetic and the fact that Excel rounds information. Behind the scenes, Excel always rounds information to 15 digits. Consider the calculation that Nick is working with:

=0.28*100-INT(0.28*100)=0

When Excel first calculates this, the precedence followed by Excel calculates it in this manner:

=(0.28*100-INT(0.28*100))=0

Note the extra set of parentheses. The result of everything to the left of that final equal sign is 3.55E-15, which means that you end up with this (longhand) formula:

=0.00000000000000355=0

This is obviously not true, which is why you get the value False returned. There are several ways to "fix" this situation. In this case, perhaps the easiest is to simply change your formula to remove the need to compare to zero:

=0.28*100=INT(0.28*100)

This formula returns True, as you would expect. This may not work for all your needs, however. So, a better rule of thumb to avoid problems is to never rely on Excel's rounding. You do this by implementing your own explicit rounding, as shown with these formulas:

=INT(0.28*100)-INT(0.28*100)=0
=ROUND(0.28*100,0)-INT(0.28*100)=0
=ROUND(0.28*100,5)-ROUND(0.28*100,5)=0

Note that you are, in these instances, not allowing Excel to perform calculations to its full precision since that can cause some unexpected results when you are comparing to a specific value, such as zero. Instead, you are forcing Excel to round the values—all the values you are working with—to whatever level of precision you need for your comparison.

Another approach is to not do the comparison to an exact value, like zero. Instead, allow for some "fudge factor" in the comparison, which allows for rounding issues. For instance, you may determine that you only care if the comparison is accurate to one one-hundredth of whatever units you are assuming. In that case, the original formula could be modified in this manner:

=0.28*100-INT(0.28*100)<0.01

This returns True, as one would expect.

ExcelTips is your source for cost-effective Microsoft Excel training. This tip (8145) applies to Microsoft Excel 2007, 2010, 2013, 2016, 2019, and Excel in Office 365. You can find a version of this tip for the older menu interface of Excel here: Avoiding Rounding Errors in Formula Results.

Author Bio

Allen Wyatt

With more than 50 non-fiction books and numerous magazine articles to his credit, Allen Wyatt is an internationally recognized author. He is president of Sharon Parq Associates, a computer and publishing services company. ...

MORE FROM ALLEN

Hyperlinks to PDF Files Won't Open

If you have a document that contains hyperlinks to a PDF file, it is possible that the links may not open the PDF file ...

Discover More

Condensing Multiple Worksheets Into One

Want a quick way to combine your worksheets? This tip features a simple macro to do the task for you.

Discover More

Automating the Importing of Macros

Macros are great when it comes to automating how you work with your workbooks. What if you want to fundamentally change ...

Discover More

Program Successfully in Excel! John Walkenbach's name is synonymous with excellence in deciphering complex technical topics. With this comprehensive guide, "Mr. Spreadsheet" shows how to maximize your Excel experience using professional spreadsheet application development tips from his own personal bookshelf. Check out Excel 2013 Power Programming with VBA today!

More ExcelTips (ribbon)

Using the MROUND Worksheet Function

If you want to round a value to some multiple of a whole number, you'll want to become familiar with the MROUND function. ...

Discover More

Rounding to Two Significant Digits

Excel provides a variety of functions you can use to round values in any number of ways. It does not, however, provide a ...

Discover More

Rounding to the Nearest $50

When preparing financial reports, it may make your data easier to understand if you round it to the nearest multiple, ...

Discover More
Subscribe

FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."

View most recent newsletter.

Comments

If you would like to add an image to your comment (not an avatar, but an image to help in making the point of your comment), include the characters [{fig}] in your comment text. You’ll be prompted to upload your image when you submit the comment. Maximum image size is 6Mpixels. Images larger than 600px wide or 1000px tall will be reduced. Up to three images may be included in a comment. All images are subject to review. Commenting privileges may be curtailed if inappropriate images are posted.

What is 2 + 8?

2019-05-15 12:33:41

Dennis Costello

If you want to dive deeper into this question, look at my posts on the "Rounding in Results" thread (https://excelribbon.tips.net/T008032_Rounding_in_Results.html). One interesting feature of this problem statement is that it involves subtracting two numbers that are very nearly identical: numerical analysis tells us that when you do that, you lose precision.

Specifically, the 0.28 was not represented exactly - it's accurate to 1 part in 2^55 (because there are 55 mantissa bits in the double-precision floating-point format. Multiplying that by 100 means the product could be off by as much as 100 parts in 2^55, which is approximately 1 part in 2^48. Subtracting 28 exactly from 28 +- 1/2^48 means we get a number that's somewhere between -1/2^48 and +1/2^48. The product turned out to be 28.0000000000000035527136788005 (1/2^48 larger than 28) - when you subtracted exactly 28 from that, the difference was far enough away from 0 that Excel wasn't willing to treat it as being equal to 0.


2019-01-02 13:20:27

Dave Bonin

I'd like to offer a small alteration of Allen's final suggestion...

Unfortunately, you cannot know whether the leftover portion will be positive or negative. Therefore, cover both bases by using the Absolute Value function:
= ABS( 0.28 * 100 - INT( 0.28 * 100 )) < 0.01


2018-12-31 22:12:53

Alex B

The 3 "=0" alternatives provided all comparing 2 identical expressions making the test a bit pointless. A better option would be to round the result as in:-
=Round(0.28*100-INT(0.28*100),5)=0
I have used 5 decimals here but anything up to 14 decimals worked in this case which seems consistent with the 15 precision limit.


This Site

Got a version of Excel that uses the ribbon interface (Excel 2007 or later)? This site is for you! If you use an earlier version of Excel, visit our ExcelTips site focusing on the menu interface.

Newest Tips
Subscribe

FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."

(Your e-mail address is not shared with anyone, ever.)

View the most recent newsletter.