The answer to this question is a combination of several facts.
Popularity
Cricket is not really a global sport (yet). Although, people of densely populated regions (Indian Subcontinent, England, South Africa) are crazy about Cricket, the sport itself has not managed to penetrate most of the countries of this world. There are only a few nations that takes Cricket seriously as compared to other popular sports like Football, Athletics, Boxing, etc.
Also, some big nations of this world are totally disinterested (or minimally interested) in this sport, e.g. USA, Japan, China, Russia, Germany, etc. which makes it even more difficult to consider for Olympics.
Arrangement
Arranging cricket games are a logistic hassle. Cricket grounds need to be specifically made; you can't hold any other sport in a cricket venue--at least not until the whole tournament is over.
Ensuring Fairness
Cricket fans know that ensuring a fare outcome is heavily dependent on the quality of the pitch. During real cricket games, host nations often prepare pitches according to the strengths of their own team, i.e. Indians will make Spinner friendly pitches while most pitches in Australia favors fast bowlers, and so on.
Thus, creating a "sporting" pitch where all participating teams will have an equal chance to win will be much challenging.
Length of The Games
Cricket matches can take a long time to finish. Until recently, the standard form of cricket game had each team bowling to the other a 50 over innings, with 6 balls per over. This means a cricket game can go on up to 4-5 hours or more.
However, the recent introduction of the shorter form of cricket (T20) makes things easier.
Lack of Adrenaline Rush
Well, this issue is debatable, but most people use it to explain why they are not interested in Cricket. Apparently, the cricket games are executed in a way which doesn't suit the "pub culture". i.e. you can't have beer and watch the games and shout out at every foul/goal attempt/goals/missed chances, etc. Instead, the games are watched in a laid back manner which is not really compatible with the theme of Olympics.
Once again, this argument is completely debatable.
Lack of Effort
No one has actually lobbied for Cricket. Not England, Not India--no one. If the cricket playing nations start lobbying, I am sure cricket will be included. However, according to David White, ICC attempted to have T20 cricket included for the 2016 Olympics but Rugby 7s and Golf were favoured in the round of voting. The ICC will try again.
I guess these are the major reasons behind cricket's absence from Olympics.
