The antithesis of fluff is meta-game. Meta-game is using the strength or weakness of the rules to build a list that performs a certain way. For example, a close combat army is a meta-game term not a ‘fluffy’ one. When you are looking at the rules you are not looking at the fluff. The fluff may justify a rule. For example Sergeant Telion has a note in his fluff that says he sometimes help train other chapters scouts to explain why he is not just a Ultramarines army option.
There are three ways to look at list based on the games fluff. Each starts with elements drawn directly from the canon. These are the fiction and background written by GW and Black Library in the 40k universe. The golden rule here is that you are basing your army list on GW’s fiction, not your own.
First, an army list based directly on a work of 40K fiction. This is the purest way to build a fluffy list. Since most books focus on a single person or unit it is rare that you can fill out a full 1500 point list. One of the best examples here is a Gaunts Ghost IG army. In this case GW has even provided rules to help. FW does a great job of building fluff and rules to go with them. The downside here is that these lists can be difficult to win with because they aren’t designed for the metagame.
Second, an army list based on the codex background. This way takes some extrapolation of how the background in the codex would work. The codex sets out how a army organizes a force. For example Space Marines have battle companies made up of 6 tactical squads, 2 assault squads and 2 devastator squads. So, in a ‘fluffy’ marine list you would have kind of the same ratio with the addition of some chapter level support. In these cases core units would always make up the bulk of the list with fewer elite or specialty units. The downside here is that it builds a boring list**.
Last is an army list with a background that you thought up, based on these first two. This is what most people intend to do when they set out to build a ‘fluffy’ list. The success of these lists as ‘fluffy’ lists is base how well you can write plausible fiction in the 40k universe. Don’t confuse this with theme or meta-game, again an all mech army is just an all mech army. The fluff is about the background and story. Don't expect anyone to understand the fluff part of your list in these cases, after all what your building is not in the cannon. These lists are just to give you focus for modeling and it is how well you do putting those elements into your army that maters.
If you like the background and fiction surrounding 40k, making it part of your army is fun. Every army needs an identity; something to hook other player in when you set them up on the table. I love playing Imperial Fists because when I lay them out on the table people know who they are and know the background. I just painted them yellow with fist icons and I have an instant connection with other player who have read the fluff. In my case I modeled them to show that they have been knee deep in siege warfare. That makes them fun. Fluff is good; it just doesn’t matter in winning games.** When I say boring, what I mean is that you will have not many elite or specialty units. I remember reading a military war gaming exercise report where the Army used the assumption that every unit they use had the same combat effectiveness as the 75th Rangers. Sure it was fun, but not an example of how the US Army is going to preform. Elite and specialty units are rare, but more fun to play.