Philip Fisher, whom Warren Buffett cites as someone from whom he learned a great deal about investing) describes five ways ordinary investors can protect themselves from being duped by a fraudulent money manager.
In 100% of fraud scams, the money manager has custody of the investor's assets. Therefore, to avoid getting scammed, Fisher's first and most important rule is that investors ensure separation between the decision-making and the custody of the assets.
This is done by placing the assets with a large, third-party institution and setting up the account such that the manager only has rights to buy and sell on your behalf i.e. he cannot withdraw funds.
Of course, this does preclude investors from buying into hedge funds or other similar structures, where investor capital is pooled as part of the fund's structure. Furthermore, it doesn't ensure that a loss of capital cannot occur; the adviser can simply make bad decisions on your behalf. However, the chances that he can make off with your money is almost nil, as long as your custodian is beyond reproach.