This is a very crude drawing of my network but hopefully it serves a purpose. I have 6 routers. One of them is the gateway to the internet. On that gateway router I have ether3 which faces the internet. Ether1 and Ether2 go to wireless bridges which go off to other towers. At each of the other towers there is an unmanaged switch that in turn goes to another tower where it connects to another switch. This process continues into a full circle back to the main gateway router.

Now at each of the towers I have a router that connects to the switch with 2 ports, ether1 and ether2 on every router. Ether1 on each router has a 10.98.0.0/28 address and each router on ether2 has a 10.99.0.0/28 address. I then created one instance and one area in ospf and put each network on that instance. Each of these 5 routers are set to redistribute connected routes always as type 1.

The main gateway router has 10.98.0.14/28 on ether1 and 10.99.0.14/28 on ether2. This router has one instance and one area setup and both networks on that instance. It is setup to redistribute default route always as type 1.

For load balancing this is working perfectly. But what I need is for 2 of the routers to go one way and the other 3 to go the other way as available bandwidth is not the same on both paths. I understand adjusting cost but if I adjust cost on the gateway router they all go one way or the other. If I adjust cost on one of the 5 routers it only affects the upload and the download still balances.

So what am I doing wrong?

Instead of using two /28s use one /30 between each two routers, but with all links in the same area. You'll still have full route distribution between all routers and thus redundancy, but can choose which path each router prefers by setting OSPF interface costs - though that won't be necessary for any of the routers other than the one at the very bottom if all your interface link speeds are equivalent. Adding cost will automatically make happen what you're trying to do. On the bottom one set a slightly higher cost on the interface you don't want to use for a default route when both interfaces are up.

Ok, I had a feeling that was my error. Let me make sure I am understanding this. If I do it that way the gateway router will still redistribute default routes as type 1, correct? But I'm going to have to redistribute more than just connected routes on the other 5 correct? Will they need to redistribute default routes also? What about redistributing other ospf routes on those?

These are in production so I need to have a firm grasp on this before I can start changing anything.

No, it's quite simpler than that.

Here a picture: All routers are in the same area (probably area 0 in case you ever add non-backbone areas). Any OSPF area can have more than one network in it. All routers in an area have the exact same link state database - if one router in an area knows about a network, so do all the others. If they have to transit other routers to reach those networks the routes are known as intra-area routes.

Red links and green links should be in different broadcast domains - meaning that you would need your switches to be VLAN capable and use one VLAN per link (probably untagged on the router uplinks), or that you would need two switches per router R2 - R6 (one for the green uplink, one for the red uplink). Though I'm not entirely sure why you need the switches at all, and aren't directly connecting the router's ethernet ports. That would simplify this by taking five or ten pieces of equipment out of the picture.

If all your interface types on all routers are the same, the cost of all links is the same. Let's say the interface cost is 10 by default to make the math easy. R1 would generate a type 1 default route via 10.98.0.1/30 with a cost of 10. R2 would learn about that. R1 would also generate a default route via 10.98.0.22/30 with a cost of 10 and R3 would learn about that. R3 would then send that default route on to R5, but there would be another interface cost of 10 in the way that is added (since it's a type 1 external route), so R5 would know about the .22 default route with a cost of 20. R5 then passes this on to R6, and so on, until R2 learns about the default route via .22 with a cost of 50. That's higher than the .1 route with a cost of 10, so R2 sends traffic back to R1 via the .1 route. This, of course, happens on all the routers - they will all have two default routes, one with a higher cost than the other.
Similarly, R1 learns about 10.103.0.1/16 behind R2 via the direct link, and also via the R3-R4-R5-R6 chain - which will have a higher cost, so R1 sends traffic via the direct link. R1 will have two routes to each network behind R2 - R6 - one with a higher cost than the other.
The only different router is R6, since if all default costs are the same, both routes will be equally good. You can either use both routes at once (something called ECMP - equal cost multipath), or you can then decide whether the red or green link is used for both upload and download traffic by setting the cost of one of the interfaces to a higher value.

Once one router distributes a route into OSPF, all the other routers in the area will learn about it. There is no need to reintroduce routes already in OSPF on any of the other routers.

Since this is on production routers on what seems to be a decent size network I would highly recommend you buy four RB750s and lab this out, leaving R4 and R5 out of your lab network (or buying 6 RB750s). They go for $40 a pop, which makes the whole thing rather affordable in a lab. That way you can prepare your entire configuration and play with it until you're comfortable, and they will serve well for future projects where you also need lab time.

Ok. This is good advice. The switches are there because I didn't really know what I was doing when I started and it was easier. They can come out. I've been considering removing them anyways. If I remove them I don't need to do anything with vlans correct?

No, because the routers will connect directly the link between them will be a dedicated broadcast domain.

While you're optimizing OSPF you may also want to stop redistributing statics. While it works many people generally consider redistribution an inferior solution when you can do it all internally to the routing protocol - and it's good practice to get more experience with OSPF.
To add the networks behind the router you would just add a network to the same area, and then add the interface and mark it as passive. That way it's natively part of the OSPF area, but won't look for neighbors. Should you need to expand beyond a router for the area (or introduce another area) you would just uncheck the passive checkbox (and possibly change network areas). This also ensures that cost propagates properly - if you were distributing statics you'd have to make sure to use type 1 so that cost gets added as the route propagates through the area. If it's a proper intranet-area LSA it all "just works".

Just to expand on all this... and possibly spark an idea for someone, here's what a typical backbone connection on my network looks like for a hybrid routed/bridged network: Management is routed, but we retain the capability to bridge VLAN traffic to/from any point on the network (MPLS/VPLS coming soon). All end-user traffic is bridged back to one of the core sites.

At the moment, we're using Cisco routers at the core site (L3 switches to be precise), I'm working out the details to migrate to a pure MT network, but we're not there yet. The core routers are connected over a ROS wireless bridge, using VLAN interfaces on a /29 network. This leaves 2 IP addresses available, which makes it super easy to bring up a new bridge when/if necessary, easing the pain of upgrades/replacements.

Fewi, Thank you! I redid the ospf exactly the way you suggested in the early morning hours and its working perfectly now!

A new question, perhaps I'm not understand what the significant of areas and instances actually do. But if I add another router, lets call it R7 and this router has the ability to connect to R5 and R6. Equal cost seems to work by default. I'm having trouble setting it up to only connect to R6 and use R5 as a backup. If I adjust cost on R7 its gateway changes to whichever has the lower cost, which is expected, but my gateway router still wants to send traffic back through R5.

When I created this router I used the built in area and instance, should I have changed the area id or anything on it?

Areas: OSPF has a strictly hierarchical design, where you separate segments of your network into areas. If there is more than one area, all those areas must connect to area 0. Area 0 is called the backbone area, and is used for transit between areas. Area IDs are 32 bit numbers expressed in dotted decimal notation, so area 0 is expressed as 0.0.0.0.
If you only have one area strictly speaking you can use any area ID you want, but it still makes sense to use 0.0.0.0 in case you ever attach other areas in the future since it's extremely likely that the initial area will be the backbone.

An instance is basically an OSPF process. Each process/instance is self contained and keeps its own tables of neighbors, routes, and link state database. To get routes from one OSPF instance to another you have to redistribute from OSPF to OSPF.

If you want to learn more about OSPF I would recommend you get the foundation learning library book for the ROUTE exam from Cisco. Even if you never plan on taking the exam the book has several chapters on OSPF that explain the theory behind it very well.

For your path choice problem for R7: R1 still sees the path to R7 as cheaper through R5 than through R6. Change the cost of the interface on R5 that connects to R7 to something that makes the alternate path cheaper. You can check "/routing ospf routes" on the routers to see what the current paths costs are.

Ok its making sense. I've got everything workign the way I want it. I created an area of 0.0.0.1 for the other router and I seem to have more control over the cost. I've got a grasp on this now so I think I can make this work. Thanks alot for all your help!