After scouring the 2010 update, the main rulebook, the 2002 annual, and even going so far as to find the old 2007 FAQ, I can find no concrete ruling.
However, if I were the manager of a campaign and this scenario came up, here's how I'd deal with it.
The main rulebook clearly states that a squadron must be an unbroken CHAIN:
Each ship must be no more than 15cm from another ship in the squadron so that the whole squadron forms a continuous chain with "links" of up to 15cm between each ship.
Bold above mine.
Because they specifically mention the word "chain" then a squadron of four consisting of two "chains" of two that are more than 15cm from each other is not in coherency (as they are not a continuous chain).
However, within each pair, the chain is continuous. With three escorts in a chain and one out, it's obvious where the "squadron" is. However, in this case it is not.
In my opinion, the owning player decides which pair of escorts constitutes "the squadron" and which pair is "out of coherency." The pair that is "the squadron" uses special orders, etc. normally. The pair "out of coherency" does not benefit from special orders until they get back into coherency by making a continuous chain with "the squadron."
Also my opinion, the player could not change their mind as to which pair was "the squadron." If they want special orders, get back into coherency!
In the case of a squadron of five that split into a pair and a trio, I'd say that the trio was "the squadron" as they constitute more of the squadron.
In a case where a character was embarked on one of the ships of the squadron (such as an Ork Warlord from the Ork Clanz list or a captain from the Battlefleet Cadia option) the escort on which the character is embarked is "the squadron" as the person giving the orders is right there! In this case, a lone escort with the character would benefit from special orders, and any other escorts in the squadron would only benefit if in a continuous chain with that specific escort. If that escort was removed for any reason, default to the opinions above.
Once again, the above calls are MY OPINION ONLY and do not hold the weight of the Main Rulebook, Armada, or the 2010 update.