Semi permanent hair dyes are gentler than the permanent ones as they do not contain peroxide found in the permanent hair dyes. It generally fades after nearly ten washes but you can remove it even earlier too if you do not like it.
Follow these simple steps to get rid of the unwanted hair color. First shampoo your hair for many times. Use warm water and a conditioner too. Then give hot oil treatment to the hair.
These kits are easily available in drug stores. You can also use virgin olive oil and leave it for 15 minutes before washing it off with water. Then coat your hair with lemon juice or vinegar for ten minutes and shampoo your hair again.
By now the semi permanent hair color should be gone but in case it persists you can repeat washing the hair.
If in any doubt or the problem persists, always consult a professional.