http://www.amazon.com/Mathematics-Elementary-Approach-Ideas-Methods/dp/0195105192/ref=tag_tdp_sv_edpp_pop_t

Try this little gem written originally by legendary mathematician, Richard Courant in 1941.   If you read only one math book in your life, this one is IT.  It is possible to absorb most of it, with just a rudimentary math knowledge, and many topics are covered, especially a wonderful approach to number theory.  It is supposedly aimed at the non-mathematician, but it goes into some depth in many areas.  At the very minimum, you'll understand binary like you never did before, and he covers that in the first chapter.