Nitrogen-doped graphene quantum dots (N-GQDs) hold promising application in electronics and optoelectronics because of their excellent photo-stability, tunable photoluminescence and high quantum yield. However, the absorption and emission mechanisms have been debated for years. Here, by employing time-dependent density functional theory, we demonstrate that the different N-doping types and positions give rise to different absorption and emission behaviors, which successfully addresses the inconsistency observed in different experiments. Specifically, center doping creates mid-states, rendering non-fluorescence, while edge N-doping modulates the energy levels of excited states and increases the radiation transition probability, thus enhancing fluorescence strength. More importantly, the even hybridization of frontier orbitals between edge N atoms and GQDs leads to a blue-shift of both absorption and emission spectra, while the uneven hybridization of frontier orbitals induces a red-shift. Solvent effects on N-GQDs are further explored by the conductor-like screening model and it is found that strong polarity of the solvent can cause a red-shift and enhance the intensity of both absorption and emission spectra.