I have personally always worked in house. But in my current job I have many workmates who came from agencies, and I can tell you they are all very happy. In general they told me they make more money and have more time. And also, coming from agencies, they have skills that people who’ve always worked in house like me lack (like, say, more experience with programmatic and that kind of stuff), and so they have added a lot of value to the team, and are very valued for it.
If you have the chance you should definitely try it.