Saturday’s man in the middle Geoff Eltringham awarded the Seagulls a dubious first half penalty which ultimately proved to be the difference between the sides.
Danny Shittu was penalised for flooring Jesse Lingard with minimal contact with David Lopez converting the resultant spot-kick.
And a perplexed Holloway remonstrated at length with the referee after the full-time whistle, although he does not expect to receive an FA fine.
“It seems like some officials are determined to be fair and impartial to the opposition,” said the Lions boss.
“If you are going to referee a game with that mindset, then you have a slight bias already, don’t you?
“I have asked the senior officials whether they have ever felt like that, because that is what it seems like to me.
“I’m glad I’m not a referee, but I know what they should do.” Holloway added: “I have promised my wife that I won’t get fined again.
“The last time I got fined was on the opening day of the season against Tottenham when a penalty was given against Dean Moxey.
“This one was worse than that.
“That cost me a fortune and this was twice, if not four times as bad as that.”
On the hotly disputed penalty, Holloway said: “In fairness to Brighton they’re a good team and their movement dragged us out of position.
“They took a quick throw-in, Danny chased him (Lingard) and I thought it was going to be a goal kick.
“It is shoulder-to-shoulder if anything.
“I can’t say that their player went down to easily because Danny is built like a proverbial.
“The referee has made a mistake.”
But despite the Lions’ latest loss, their fourth in six games, Holloway is confident his side will get themselves away from relegation danger.
He explained: “We’re in a bad situation but I’m still so confident that we will turn this around.
“The tougher it gets, the tougher we have to get.
“We will keep fighting right until the end because that’s our duty.
“I truly believe we will have been better than three other teams by the end of the season, and that’s what will matter.”
The Den chief added: “Everybody has to keep believing.
“On Saturday we didn’t get anything near what our performance deserved.
“I saw some spirit, and that’s more like it.
“The effort from everybody was unbelievable and I think you could see that the fans, who were brilliant.”
Follow us on Twitter @NewsShopperSprt for live updates from Millwall’s next two games away at Derby and Blackpool