The story behind the new file system is twofold.
On the one hand, and as you already know, Moodle calculates the SHA1 hash of the file content and uses this hash to rename the file and to create the two-level directory path (with the first 4 characters of the hash) where it will be saved.
For example, if the hash of a file is "96d142ae...", then the path "96/d1" will be created under moodledata/filedir, like so: moodledata/filedir/96/d1.
This hash is saved in the contenthash field of the mdl_files table.
On the other hand, Moodle creates another SHA1 hash based on the pathname (or context) in which the file is being used (inside Moodle). Specifically, this hash is created based in the following info: contextid, component, filearea, itemid, filepath and filename.extension.
For example, if a file named bgd1l1.swf is located under contextid=24, component=mod_scorm, filearea=content, itemid=0, and filepath="/data/swf/", then Moodle will calculate the SHA1 hash for the string "/24/mod_scorm/content/0/data/swf/bgd1l1.swf".
This hash is saved in the pathnamehash field of the mdl_files table.
My guess is that the incorrect pool file content error you had was because eventhough the contenthash value of both, the 900 MB and the 1.6 GB files was different, they actually had the same pathnamehash value, something that is not allowed because that is a uniquely indexed field. And so, that is why you first had to delete the 900 MB file.
Hope this helps.