As I discussed above, today search engines rank pages using a combination of off-page and on-page factors. Additionally there are two types of factors, quality modifiers and topic modifiers.
On-page factors are anything that is on your page, not just what is visible, but anything in the source code search engines can see. To see what search engines see you can browse to your page and if using Internet Explorer go View>Source or, if using Mozilla Firefox, go View>Page Source. This is what the search engines see.
Off-page factors are primarily links and the context associated with them. Remember that I'm saying off-page, and not off-site. Even though you look at your site as one whole, search engines are much more interested in your pages as individuals. Think of links as votes, a link from one page to another is a vote from one page to another. The more votes you have, the better. Additionally the anchor text, the text used within the link, is part of the equation in that it indicates what the vote is for. The text around the link, the topic of the page in general, and every who links to the page that links to you, all of these can modify how much benefit you will get from each link.
Quality modifiers are factors that actually make the difference between being ranked #20 and being ranked #1. They are factors which search engines take to mean that your site is good. The biggest quality factors are off-page ones as search engines no longer trust webmasters to rate their own quality. So incoming links are the main quality factor in search engine algorithms. Also, many spam filters and or penalties are for your overall quality, there are a lot of little things that can bring you down in this regard.
Topic modifiers are factors that help a search engine to decide which keywords to rank your page under (as opposed to how high or low to rank it). The main topic modifiers are on-page factors such as keyword usage on your page, your title, your headings, and other on-page factors that will be discussed in other articles. However, the off-page context of your incoming links is a very important topic modifier as well.
The point I want to make with these factors is that you need all of them in a good balance in order to be successful in your search engine optimization efforts. A million incoming links but poor on-page optimization and you'll not rank well. Perfect on-page optimization but no incoming links and you'll not rank well. Good on-page optimization, lots of incoming links, but bad context & anchor text for your incoming links and you'll not rank well. You need as many incoming links as you can get, but within a good context for your website, and you need good on-page optimization without breaking any search engine guidelines. Do all that and you will rank well.
Maybe people mistakenly believe search engines want your website to be accessible and perfectly coded. This is not true. The issue is that building an accessible or usable site and building an effectively optimized site are coinciding goals for the most part. Many of the same changes you would make to enhance the usability of your site also help you rank better in the search engines, they help in each way for different reasons but they do help in each way.
Then of course, as I mentioned above, anything that makes a site of a higher quality generally earns that site more incoming links, so building a usable site is certainly a good idea, just make sure you understand that there isn't any direct bonus for usability, it is easy to get carried away with it if you make that mistake.