Please Note: This article is written for users of the following Microsoft Excel versions: 2007, 2010, 2013, and 2016. If you are using an earlier version (Excel 2003 or earlier), this tip may not work for you. For a version of this tip written specifically for earlier versions of Excel, click here: Avoiding Rounding Errors in Formula Results.

# Avoiding Rounding Errors in Formula Results

by Allen Wyatt
(last updated November 2, 2017)

Nick notes that the formula "=0.28*100-INT(0.28*100)=0" returns False even though it is obviously true. He believes the issue has to do with rounding and how the computer uses binary arithmetic, etc. Nick is using the formula as a part of a larger IF statement, and he assumes there are other rounding errors in Excel that can reach out and bite him. He wonders if there is a simple way to avoid formula pitfalls such as this.

Before looking at simple ways to avoid this type of problem, it is important to understand why the problem exists. As Nick notes, it really has to do with binary arithmetic and the fact that Excel rounds information. Behind the scenes, Excel always rounds information to 15 digits. Consider the calculation that Nick is working with:

```=0.28*100-INT(0.28*100)=0
```

When Excel first calculates this, the precedence followed by Excel calculates it in this manner:

```=(0.28*100-INT(0.28*100))=0
```

Note the extra set of parentheses. The result of everything to the left of that final equal sign is 3.55E-15, which means that you end up with this (longhand) formula:

```=0.00000000000000355=0
```

This is obviously not true, which is why you get the value False returned. There are several ways to "fix" this situation. In this case, perhaps the easiest is to simply change your formula to remove the need to compare to zero:

```=0.28*100=INT(0.28*100)
```

This formula returns True, as you would expect. This may not work for all your needs, however. So, a better rule of thumb to avoid problems is to never rely on Excel's rounding. You do this by implementing your own explicit rounding, as shown with these formulas:

```=INT(0.28*100)-INT(0.28*100)=0
=ROUND(0.28*100,0)-INT(0.28*100)=0
=ROUND(0.28*100,5)-ROUND(0.28*100,5)=0
```

Note that you are, in these instances, not allowing Excel to perform calculations to its full precision since that can cause some unexpected results when you are comparing to a specific value, such as zero. Instead, you are forcing Excel to round the values—all the values you are working with—to whatever level of precision you need for your comparison.

Another approach is to not do the comparison to an exact value, like zero. Instead, allow for some "fudge factor" in the comparison, which allows for rounding issues. For instance, you may determine that you only care if the comparison is accurate to one one-hundredth of whatever units you are assuming. In that case, the original formula could be modified in this manner:

```=0.28*100-INT(0.28*100)<0.01
```

This returns True, as one would expect.

ExcelTips is your source for cost-effective Microsoft Excel training. This tip (8145) applies to Microsoft Excel 2007, 2010, 2013, and 2016. You can find a version of this tip for the older menu interface of Excel here: Avoiding Rounding Errors in Formula Results.

##### Author Bio

Allen Wyatt

With more than 50 non-fiction books and numerous magazine articles to his credit, Allen Wyatt is an internationally recognized author. He  is president of Sharon Parq Associates, a computer and publishing services company. ...

##### MORE FROM ALLEN

Removing Blank Pages at the End of Your Document

You go to print out your document, and all of a sudden notice that there was a blank page that printed at the end. This ...

Discover More

Recording a Macro

One of the most common ways of creating macros is to use Excel's macro recorder. This tip shows how easy it is to use the ...

Discover More

Customization of Word is done, largely, through what you add to or remove from the Quick Access Toolbar. When you have ...

Discover More

Excel Smarts for Beginners! Featuring the friendly and trusted For Dummies style, this popular guide shows beginners how to get up and running with Excel while also helping more experienced users get comfortable with the newest features. Check out Excel 2013 For Dummies today!

##### More ExcelTips (ribbon)

Rounding Religious Wars

How to round values half way between two boundaries is apparently a topic of contention in some sectors of the ...

Discover More

Rounding To the Nearest Even Integer

Do you need your numbers to be rounded to an even integer value? How you accomplish the task depends on the nature of the ...

Discover More

Rounding Up to the Next Half

When processing data it is not unusual to need to round that data in some way. For instance, you may need to round a ...

Discover More
##### Subscribe

FREE SERVICE: Get tips like this every week in ExcelTips, a free productivity newsletter. Enter your address and click "Subscribe."

If you would like to add an image to your comment (not an avatar, but an image to help in making the point of your comment), include the characters [{fig}] in your comment text. You’ll be prompted to upload your image when you submit the comment. Maximum image size is 6Mpixels. Images larger than 600px wide or 1000px tall will be reduced. Up to three images may be included in a comment. All images are subject to review. Commenting privileges may be curtailed if inappropriate images are posted.

What is 5 + 2?

2017-10-12 10:39:39

Dave Bonin

The formulas Allen proposed are rarely quite that simple. Typically, some of the constants
would be replaced by cell contents. I man, why else bother?

To illustrate, lets adjust the proposed formula
= 0.28 * 100 - INT( 0.28 * 100 ) < 0.01
to replace a constant with a cell:
= B7 * 100 - INT( 0.28 * 100 ) < 0.01

This formula has a small issue -- namely that it is true for tiny positive values and all negative
values, including very large negative values like -87,000.

For this reason, I like to include the ABS() function
= ABS( B7 * 100 - INT( 0.28 * 100 )) < 0.01

This properly captures our intent without worrying about signs and such.

Also, I typically set the comparison constant, eg: 0.01, to a number that is
two digits smaller than the math I'm concerned about. If my level of interest
is integers, I'd use 0.01. If I was interested in a hundredth of a percent, I'd use
0.000001.

Not sure what comparison constant to use? Look at the number formats of the
cells you're using and go two digits finer.

2017-10-12 08:36:32

Terry

When i studied computer science, it was a rule that you NEVER compared two floating point numbers in an IF statement to be exactly equal. We always used the fudge factor approach, as: IF( ABS(x-y)<1.e-6) THEN .... where x and y are some computed real (floating point) numbers. Due to round off, two real numbers are very unlikely to be exactly the same... Also,the absolute value is needed because you don't know the sign of the subtraction result.

2015-07-29 10:01:09

ong\'aro Moses

Thanks guys, I find this group very helpful.

2013-10-07 08:51:27

Bryan

This is the second time I've found an article that I know I've commented on, where comments have been deleted. As recently as 9/24/13, the following comments were on this article:

-----------------

Bryan 30 Jun 2013, 09:13
Very true, but since we are talking about Excel if I had meant factorial it would have been =0.28*FACT(100) ;-)

-----------------

Greg 28 Jun 2013, 09:43
Bryan be careful with your exclamation marks: .28*100! means .28*Factorial 100 and as factorial 100 is 93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000 thus you could expect to see a rounding error :-)

-----------------

Bryan 28 Jun 2013, 07:47
This is definitely unexpected behavior. Even with extra precision, you wouldn't expect to ever see a rounding error with .28*100!

Another way to avoid this error (especially if you can't use the INT or ROUND functions for data integrity reasons) is to separate the calculations into two separate cells. So, if A1 = 0.28*100-INT(0.28*100), then in cell B1 the formula =A1=0 returns TRUE.

I can't explain why, but it works.

-----------------
Source:
http://cc.bingj.com/cache.aspx?q=Avoiding+Rounding+Errors+in+Formula+Results&d=4530422873524231&mkt=en-US&setlang=en-US&w=_oZRDCzaNpsMMsvnPbGNYx4zGfxcvK00

##### This Site

Got a version of Excel that uses the ribbon interface (Excel 2007 or later)? This site is for you! If you use an earlier version of Excel, visit our ExcelTips site focusing on the menu interface.