Tunnel over IPSec While Allowing Connections to WAN IP

So I have two RB2011s at different locations. I have the following setup.

Site 1 - WAN 1.1.1.1
LAN 10.0.0.1
IPIP 172.16.0.1 (Local 1.1.1.1, Remote 1.1.1.2)
IPSec Policy (Dst 1.1.1.2, Src 1.1.1.1, Protocol All)

Site 2 - WAN 1.1.1.2
LAN 10.0.1.1
IPIP 172.16.0.2 (Local 1.1.1.2, Remote 1.1.1.1)
IPSec Policy (Dst 1.1.1.1, Src 1.1.1.2, Protocol All)

Basically everything seems to work, I can ping both ends of the tunnel, and get the IPSec to come up. My issue is that if I am at site 1, I can no longer ping 1.1.1.1. If I disable the IPSec policy it works again. Is there any way to fix that? Is there a way I can only IPSec the tunnel and not break access via the WAN interface? Does that make sense?

Thanks.
-Eric

As a follow up to this…

Would it be reasonable to just set the policy on each end to only IPSec Protocol #47? … that way only Tunnel related traffic goes over IPSec… everything else goes direct. Or would that leave it open to vulnerabilities.

I tried setting it up that way and it works correctly, I am just not sure if I am not encrypting something I should be by only setting it to protocol 47.