One of the key aspects of Ferdinand de Sassure's theory and of structuralism is the notion of the arbitrariness of the sign. In fact, Saussure stressed the arbitrariness of the sign as the first principle of semiology I (the study of signs which includes linguistics). By saying that signs are arbitrary, Saussure was saying that there is no good reason why we use the sequence of sounds 'sister' to mean a female sibling. We could just as well use 'soeur', 'Schwester', 'sorella'. For that matter, we could just as well use the sequence of sounds: 'brother'. Of course, as he pointed out, we don't have any choice in the matter. If we want to talk about female siblings in the English language, we can only talk about 'female siblings' or 'sisters'.
The point of the arbitrariness of the sign is that there is not compelling necessary conncetion between signifier and signified, and therefore language as a system determines meaning which does not originate outside of language. Saussure saw language as being an ordered system of signs whose meanings are arrived at arbitrarily by a cultural convention.
Understanding De Saussure's nature of the linguistic sign can lead us to undersatnd why the source of meaning for him is difference. Arbitrariness and difference go together since there is no positive bond between a signifier and a signified, only the relative position of the that bond withing the system of language.
For more on the nature of the sign an de Saussure's thought: