We report a study of the suppressed decays B--->[K(+)pi(-)](D)K- and B--->[K(+)pi(-)](D)pi(-), where [K(+)pi(-)](D) indicates that the K+pi(-) pair originates from a neutral D meson. These decay modes are sensitive to the unitarity triangle angle varphi(3). We use a data sample containing 275 x 10(6) BB pairs recorded at the Upsilon(4S) resonance with the Belle detector at the KEKB asymmetric e(+)e(-) storage ring. The signal for B--->[K(+)pi(-)](D)K- is not statistically significant, and we set a limit r(B)<0.27 at 90% confidence level, where r(B) is the magnitude of the ratio of amplitudes |A(B--->D 0K-)/A(B--->D0K-)|. We observe a signal with 6.4sigma statistical significance in the related mode, B--->[K(+)pi(-)](D)pi(-).