Short response: entering the Metro system will count you one trip, no matter how many transfers between Metro lines you do. Transferring from Metro to a bus, or from one bus to another one, will count you two trips.
Long response (and probably most of the tourists who come to Madrid don't care): Metro is divided into three separated networks regarding fares:
- Line 9 from Puerta de Arganda to Arganda del Rey
- The whole line 12, plus Joaquin Vilumbrales and Puerta del Sur stations in line 10
- Any other station not included above.
The last one is officially called MetroMadrid, but is what everybody understands as Metro, and its fares work as mentioned in this thread.
The first network is operated by another company called TFM, it has their own tickets and MetroMadrid ones are not valid in this net.
Same happens to the first one, called Metrosur, that has its own ticketing system.
The only exception to this is the "Combined ticket", a ticket that can be purchased (1.50 EUR) in any station of any network, and allows the owner to transfer between networks.