This paper considers a simple model of credit risk and derives the limit distribution of losses under different assumptions regarding the structure of systematic risk and the nature of exposure or firm heterogeneity. We derive fat-tailed correlated loss distributions arising from Gaussian risk factors and explore the potential for risk diversification. Where possible the results are generalised to non-Gaussian distributions. The theoretical results indicate that if the firm parameters are heterogeneous but come from a common distribution, for sufficiently large portfolios there is no scope for further risk reduction through active portfolio management. However, if the firm parameters come from different distributions, then further risk reduction is possible by changing the portfolio weights. In either case, neglecting parameter heterogeneity can lead to underestimation of expected losses. But, once expected losses are controlled for, neglecting parameter heterogeneity can lead to overestimation of risk, whether measured by unexpected loss or value-at-risk