Inbreeding occurs when an individual has one or more common ancestors. A common ancestor is one that is present on both sides of the pedigree. I.e., in order for an individual to have a non-zero coefficient of inbreeding it is necessary for both father(sire) and mother(dam) to be descended from one or more common ancestors.
The extent of inbreeding is commonly expressed using a formula first derived by the American geneticist Sewall Wright early in the 20th century and for this reason it is often known as Wrights Coefficient of Inbreeding. Traditionally the symbol F is used to signify the coefficient of inbreeding.
Consider for one moment the underlying genetic basis of heredity. With most genes, any individual has two copies of each gene - two 'alleles'. One copy is received from the mother, the other from the father. Now examine what can happen when both mother and father have a common ancestor e.g. they have the same father. This could be represented by the following pedigree:
D | B---| | | | E A---| | D | | C---| | F
I.e. the father B and mother C of individual A both have the same father D. Or, to put it another way, individual A has a common grandfather, D.
Now consider any gene in D. That gene consists of a pair of alleles. One of these alleles will be passed from D to B, and there is a 50% chance that B will pass this allele on to A. Separately, there is a 50% chance that D will pass this same allele to C. Likewise there is also a 50% chance that C then passes this allele to A. Therefore there must be a 50% x 50% x 50% = 12.5% chance that A has two copies of the same allele - inherited from the common ancestor D. This value is what we mean by the coefficient of inbreeding - it is the probability that any two alleles at a given gene locus are the same allele.
Now consider what would happen if we changed the pedigree as follows:
D | B---| | | | E A---| D | G---| | | H C---| | F
I.e. D is now a grandfather of C rather than a father. The inbreeding will be halved, to 6.25% because the likelihood of the same allele being inherited via C is halved.
Consider further what happens when no ancestor appears on both sides of the pedigree - F drops to zero. I.e. Outcrossing to a completely unrelated individual always reduces F to zero at a stroke.
These are very simple examples. In real life, within a closed population things are often very much more complicated because the same individuals can appear in several places on both sides of the pedigree, and you can often have multiple common ancestors. The principle behind the calculation remains the same - each probable source of inbreeding has a cumulative effect on the overall value.