Some people aren't remotely worried about designer babies. It comes down to being responsible but the benefits of genetically modifying babies could be huge. Imagine a world where there are no longer genetic diseases. You can inherit all the good things from your parents, and none of the nasty things—like a predisposition for diabetes, depression or any other number of things genetics determine.
If some parents want specific things from their child, so be it. It's their child and it's not up to anyone else to decide. Government officials and scientists should take the role of setting guidelines to ensure the safety of the child and our population. But if a parent really wants their child to be tall, or have a certain hair color, they should be allowed to do so. It doesn't harm anyone.
Designing babies is already here. With IVF, parents are able to choose which embryo they want to implant. That essentially accomplishes everything that "designing" babies would. The fears with designer babies are overblown. They'll ultimately be good for the world.
Others think creating new humans should be left to nature. There are a lot of things we still don't know about biology and genetics. Tampering with the fundamental building blocks of humanity will not end well. It's wrong on an ethical level and a potential disaster on the biological level.
In all likelihood, the first people who will have access to the technology that would create designer babies will be rich and well connected. This literally creates a separate class of people that will not only be advantaged socially and economically, but physically and genetically. It's easy to see how quickly society could morph into a world of haves and have nots.
At a biological level, people worry about the unintended consequences of designing babies. Selecting for cosmetic changes, like hair or eye color, could lead to a snowball effect in regards to personality, intelligence or any number of things. Humans evolved over time by responding to pressures from their environment. Instead of naturally evolving, humans would be taking evolution into their own hands and escalating it at warp speed.