Couldn't find info via search function. Someone tell me what the factory speaker sizes are for SW20?
 
Apparently it's 3.5 rear 4 front, but just want to confirm.
If so what are the biggest speakers you can fit without modifying the speaker/bracket, as it's pretty hard to find 3.5inch speakers now.

Thats right, from what i remember:
Stock OEM
F: 4"
R: 3.5"
 
I have on the front.
F: 6.5"
They are ALPINE SPS-610C -G and 6.5" and depth of 2-1/4" with no modifications to the speaker bracket. Depending on the depth of the woofer it may come in contact with the window glass when fully retracted. But i have no issues with my current setup.

Would 4 inch fit in the rears?

I dont know. Dont have any speakers in the rear. Cars stripped in the rear with a rollcage. Sorry.