Skip to content

Instantly share code, notes, and snippets.

@charterchap
Created September 16, 2015 03:30
Show Gist options
  • Save charterchap/8c4250a7e0d73891a1fc to your computer and use it in GitHub Desktop.
Save charterchap/8c4250a7e0d73891a1fc to your computer and use it in GitHub Desktop.
atomic writes linux c++, QT, Boost, GNU
// Practical snippets for loosely atomic writes on linux
// Important, even more so on allocate on flush file systems such as xfs and ext4.
// You can learn more here: https://www.flamingspork.com/talks/
// https://www.kernel.org/doc/Documentation/filesystems/ext4.txt
// These snippets are not perfect, if writing cross platform applications - marco guards
// Just trying to convey the gist
// The basic routine is:
// 1. Write data as file.new
// 2. write data
// 3. flush user space buffers
// 4. sync from kernal space to hardware
// 5. hope the hardware does what you want
// 6. rename the file, rename is basically atomic (in posix land)
//
// the close to c++ way:
assert(__GNUG__); //gnu c++ specific code
//setup ofstream with an fsync friendly filebuf for those allocate-on-flush
// filesystems (xfs,ext4,etc)
const std::string filename_tmp(filename+".new");
FILE *fp;
fp=fopen(filename_tmp.c_str(), "w");
//to fsync when using ofstream
__gnu_cxx::stdio_filebuf<char>
linux_fb(fp, std::ofstream::out | std::ofstream::trunc);
std::ofstream os;
os.std::ios::rdbuf(&linux_fb); //associate ofstream with gnu basic_filebuf
//the ofstream methods will not behave as expected from here out.
assert(linux_fb.is_open());
if (linux_fb.is_open()==false){
//fail or whatnot
}
//... use the ostream to write data
if(sysconf(_POSIX_FSYNC)) //or use as a predefined macro
{
assert(linux_fb.fd());
if(linux_fb.fd())
{
// close (per section 27.8.1.3 c++ std) will flush characters
// and disassociate linux_fb from ofstream
os.close();
if(os.good())
{
fsync(linux_fb.fd()); // sync to hardware
}
}
}
else
{
os.close();
}
linux_fb.close();
int fail_status;
fail_status = rename(filename_tmp.c_str(), filename.c_str());
if(0 != fail_status)
{
//fail
}
//
// using boost iostreams:
#include <boost/iostreams/device/file.hpp>
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/iostreams/stream.hpp>
namespace io = boost::iostreams;
std::string fileName_tmp = fileName + ".new";
io::file_descriptor_sink saveFile( fileName_tmp, std::ios_base::out );
io::stream_buffer<io::file_descriptor_sink> stream(saveFile);
std::ostream toFile(&stream);
toFile << "Hello, my tuppentup friend";
if(sysconf(_POSIX_FSYNC))
{
toFile.flush();
if(toFile.good()) //if flush did not fail
{
fsync(saveFile.handle());
}
}
stream.close(); //close file
rename(fileName_tmp.c_str(), fileName.c_str());
//
// Qt QFile (some serious limitations):
std::string fileName_tmp = fileName + ".new";
QFile outFile(fileName_tmp.c_str());
if ( outFile.open(QIODevice::Truncate | QIODevice::WriteOnly) )
{
QTextStream outStream( &outFile );
outStream << "hello, my tuppentup friend";
if(sysconf(_POSIX_FSYNC))
{
outStream.flush();
if(outFile.flush())
{
fsync(outFile.handle());
}
}
outFile.close();
// See ksavefile.cpp from KDE project for more about QT rename shortcomings
outFile.rename(fileName.c_str()); // This is not atomic like a POSIX rename
}
// One could also macro guard with #if _POSIX_FSYNC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment