Now, I don't really know of any rules so I thought why not try to derive some. Looking at some some of the words that have ന or ന്, I came up with the heuristics:
- When it begins a word, ന is pronounced Na. e.g. നിലാവ്, നുള്ളി, നൃപന്, നെടുകേ
- When it occurs as a simple letter (i.e. a non-compund letter), it is pronounced na. e.g. കനല്, കനി, മനുഷ്യന്, അങ്ങനെ,
- When it occurs as the chillu (ന്) it is always pronounced n
These heuristics seem to work even for complex cases like നാനാനനന് (Naanaananan - ബ്രഹ്മാവ്).
The heuristics for complex letters don't seem as simple. Let us look at some examples:
- തിന്നുക is nnu, തിന്നു is NNu
- പന്ന is nna, പന്നി is NNi
- നിമ്നം is na, നഗ്നം is Na
So, there doesn't seem to be a rule.
I am tempted to say that words of Sanskrit origin use na whereas Na is used only in words of Tamil origin. But this also doesn't explain the Tamil words with pronunciation of na.