OMG!
I figured it out finally.
This post made it clear, after a thorough reading:
http://forum.mikrotik.com/t/possibility-of-multiple-trunked-vlan-master-ports-on-single-crs125-switch-chip/95398/1
Basically DO NOT use multiple master ports if you want to do VLANs.
This super easy to miss note from http://wiki.mikrotik.com/wiki/Manual:CRS_examples sort of explains it:
Note: Multiple master-port configuration is designed as fast and simple port isolation solution, but it limits part of VLAN functionality supported by CRS switch-chip. For advanced configurations use one master-port within CRS switch chip for all ports, configure VLANs and isolate port groups with port isolation profile configuration.
Here is my working config. And by working I mean I can ping 192.168.1.16 (the CRS) from my Linux box (port 22) and also SSH into the CRS from ports 17-24 without the Linux box being aware of the VLAN.
Ports 9-16 are setup to be tagged as VLAN 100. Tested and working.
Ports 17-24 are setup to be untagged VLAN 100. So VLAN tags are not seen by the connected devices, but traffic will get put onto VLAN 100. Tested and working.
This is not a fully complete setup. VLAN isolation should be turned on and maybe other things.
# feb/29/2016 20:37:17 by RouterOS 6.32.4
# software id = 36UD-T26V
#
/interface ethernet
set [ find default-name=ether2 ] master-port=ether1
set [ find default-name=ether3 ] master-port=ether1
set [ find default-name=ether4 ] master-port=ether1
set [ find default-name=ether5 ] master-port=ether1
set [ find default-name=ether6 ] master-port=ether1
set [ find default-name=ether7 ] master-port=ether1
set [ find default-name=ether8 ] master-port=ether1
set [ find default-name=ether9 ] master-port=ether1
set [ find default-name=ether10 ] master-port=ether1
set [ find default-name=ether11 ] master-port=ether1
set [ find default-name=ether12 ] master-port=ether1
set [ find default-name=ether13 ] master-port=ether1
set [ find default-name=ether14 ] master-port=ether1
set [ find default-name=ether15 ] master-port=ether1
set [ find default-name=ether16 ] master-port=ether1
set [ find default-name=ether17 ] master-port=ether1
set [ find default-name=ether18 ] master-port=ether1
set [ find default-name=ether19 ] master-port=ether1
set [ find default-name=ether20 ] master-port=ether1
set [ find default-name=ether21 ] master-port=ether1
set [ find default-name=ether22 ] master-port=ether1
set [ find default-name=ether23 ] master-port=ether1
set [ find default-name=ether24 ] master-port=ether1
set [ find default-name=sfp1 ] master-port=ether1
/interface vlan
add interface=ether1 name=vlan100 vlan-id=100
/interface ethernet switch egress-vlan-tag
add tagged-ports=ether9,ether10,ether11,ether12,ether13,ether14,ether15,ether16,switch1-cpu vlan-id=100
/interface ethernet switch ingress-vlan-translation
add customer-vid=0 new-customer-vid=100 ports=ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24 sa-learning=yes
/interface ethernet switch vlan
add ports=ether9,ether10,ether11,ether12,ether13,ether14,ether15,ether16,ether17,ether18,ether19,ether20,ether21,ether22,ether23,ether24,switch1-cpu vlan-id=100
/ip address
add address=192.168.1.16/24 interface=vlan100 network=192.168.1.0