James Gray become embroiled in a race row after he was reported to have made the derogatory remark at an event for St John’s Ambulance, where both ministers were guests.
The 66-year-old MP denied denies the quote attributed to him that “they all look the same to me” – but admitted saying he thought the ministers look alike.
Condemning his remarks, a Conservative Party spokesperson said: “These comments were misjudged. We do not tolerate racism or discrimination of any kind.”
The Independent understands that Mr Gray has been reminded by Tory chief whip Mark Spencer of the standards expected of him as an MP – but no further action has been taken.
St John’s Ambulance was swift to take action over the comments, asking the MP to step back from his role as a parliamentary advocate for the ambulance service.
A spokesperson for the charity said: “St John does not tolerate racism in any way, shape or form. We spoke with James Gray following the event about our values as an open, inclusive and progressive charity.”
Labour also condemned his reported remarks, and challenged Boris Johnson to take “firm” disciplinary action against Mr Gray.
Taiwo Owatemi MP, Labour’s shadow minister for women and equalities, said: “Enough is enough. These reported remarks from James Gray are shameful and completely unacceptable.”
She added: “Boris Johnson needs to change his habit and take firm action here, making clear that racist language has no place in a mainstream political party. These remarks simply cannot go unchallenged.”
An eyewitness at the recent parliamentary event told the MailOnline – which first reported the comments – that Mr Zahawi immediately had a private conversation with Mr Gray following his speech.
But he denied he was spoken to by Mr Zahawi after the incident, and said says he is “very close friends” with both ministers, who are among the most high-profile politicians in Westminster.
He told the MailOnline: “The notion that this is some sort of racist remark is ridiculous. They are two very good friends of mine. I said, ‘I am sorry to confuse the two of you. You two look very alike.’ I said, ‘I am sorry if I got you two mixed up.’”
Mr Gray was forced to apologise only last month for a “foolish remark” suggesting a bomb should be planted in a senior Labour MP’s office.
At the time Mr Gray said he meant “no offence” with the comment about Labour chair Anneliese Dodds, which that he posted in a WhatsApp group ahead of the opposition’s conference in Brighton.
The timing of the remark particularly raised concern among MPs because it was at the Conservative Party conference in the same city in 1984 that Margaret Thatcher was targeted by a bombing.