I assume you have converted to HSV or HSI colour space.
The colour (e.g. blue rather than yellow, red etc) is in the Hue component.
Saturation encodes how strongly coloured a pixel is, not what colour it is.
For low values of intensity saturation values are undefined (noise).
For low values of saturation Hue is undefined (noise). You may need to Threshold first for I>I_min then S>S_min then for H_low<H>H_hi.
You could examine pixel values in each component for pixels of your target colour.
You could code adjustable thresholds and vary the values until you get a suitable result.