When working with checkbox fields and formulas, "= 0" means that a box has not been checked, and "= 1" means that a box has been checked.
So, to make a formula field which returns A when a box has been checked, and B otherwise, you would need to set the formula for the formula field as:
IF(Checkbox = 1, "A", "B")
If you'd like to see how to use a Rollup field to count all the checkboxes within a field, check out this Community post!
How do I return a checkbox as the result of a formula?
If you are looking to return a checkbox as the result of a formula, you won't be returning a checkbox field, but rather a checkbox emoji✅ . Check out this support article for information about writing formulas with emojis.