Well, boarding actions have people going across to both ships no? Besides, the vessel would no doubt be unleashing a number of toxins and plagues onto the subject vessel.
Let's look at how much of a penalty it is... (quite important)
So results on 2D6 pick the lowest are:
1,1 1,2 1,3, 1,4, 1,5, 1,6 2,1 3,1 4,1 5,1 6,1 =1 (11)
2,2 2,3 2,4 2,5 2,6 6,2 5,2 4,2 3,2 = 2 (9)
3,3 3,4 3,5 3,6 6,3 5,3 4,3 =3 (7)
4,4 4,5 4,6 6,4 5,4 = 4 (5)
5,5 5,6 6,5 =5 (3)
6,6 = 6 (1)
Overall average roll. ~2.5
Compared to a normal average of 3.5
Not terribly harsh.