Factories are great when your business logic does not really depend on complex objects relationships and setup.
Fixtures are more powerful when you need to setup a complex object architecture with very specific attributes. For instance, if you want to test a tax engine that depends on a set of transactions with very specific attributes (currencies, amounts, etc.), a set of users with very specific attributes (citizenship, business status, etc.), it will be easier to write this setup using fixtures.
client:
fields:
name: Charles
client_french:
inherit_from: client
fields:
citizenship: french
...
You can have set values in factories or build them with the case in mind :) IMO.