You can only retarget people that Facebook recognizes.
So if a person visits your website with the same combination of browser, device and other criteria than with which they visit Facebook’s website, the pixel will be able to match the “visitor” with this persons Facebook user ID.
So correct, the bad news is this means you cannot retarget anyone who visited your website, only the ones that Facebook can match with one of their users. The good news are I think it’s safe to say that Facebook is pretty good at matching visitors with their users 😉