It's got to do with these 2 things.
It's fairly common to hear someone say something like "You two are like chalk and cheese". It doesn't make a ton of sense if you think about those kinds of chalk and cheese, though.
It helps if you keep in mind these versions.
So now do you see? It's similar to the American expression "comparing apples and oranges", but I would argue even better.
1. It's alliterative. Alliteration is fun and cool.
2. It's adds a whole level of nuance that "apples to oranges" misses
So I guess the reasoning of "apples to oranges" is that they are both in the same general category, so it is unfair to compare to things that are not equal.
"Chalk and cheese" takes it to a different place. It implies that you are dealing with two things that, at first glance, might be confused. The two things are very different and serve entirely different purposes, so they are in no way alike once you look close enough to get past the initial similarities.
Kinda cool huh?
I wonder how many British people use this phrase without actually understanding why it means what it does?
I bet a lot of them are picturing sidewalk chalk and cheddar!