The phrase "only problem" is an idiomatic expression that means there is a single issue or difficulty, but it may not be seen as significant compared to other more pressing matters. It can also imply that despite this one challenge, everything else seems to be going well and there are no major obstacles in the way.