When working with checkbox fields and formulas, "= 0" means that a box has not been checked, and "= 1" means that a box has been checked.

So, to make a formula field which returns A when a box has been checked, and B otherwise, you would need to set the formula for the formula field as:

IF(Checkbox = 1, "A", "B")