Elizabeth Arden blamed Justin Bieber's and Taylor Swift's poor fragrance sales for a significant drop in revenue and stock prices.