Next generation cellular network is expected to overcome the issues like demand for high data rates, better network coverage, energy, and spectrally efficient system. In this paper, femto based heterogeneous cellular network (HCN) with cognitive approach is considered to address these issues. This paper investigates two tier cognitive based HCN for overlay, underlay, and mixed overlay-underlay spectrum sharing schemes (SSSs). Macro base station (MBS) and cognitive femto base station (CFBS) are considered as a primary user network (PUN) and cognitive user network (CUN), respectively.

Outage probability is a fundamental performance measure in HCN design. In this work, stochastic geometry is used for deriving the probability density function (pdf) of the signal-to-interference ratio (SIR) and the outage probability of the HCN for different SSSs. Furthermore, energy efficiency (EE) of the HCN and single tier network (MBS only) are also calculated for different SSSs. Numerical results show that outage performance of HCN in mixed overlay-underlay scheme outperforms other schemes. Moreover, it is observed that the effect of BSs density on outage probability is lower for the CFBSs as compared to MBSs.