Musings on the life and times of the STLSoft libraries

Scoping MSVCRT memory tracking flags with scoped_handle

2010-12-28T21:24:00.000-08:00

Am just working on some diagnostic extras to be provided as a side-project for Pantheios, with a particular focus on main(). In applying them to some of my system tool programs, I'm finding some false positive memory leaks being reported for FastFormat: this is because FastFormat caches parsed format strings, and they're not released until the library is uninitialised.

I want to "hide" these memory allocations from Visual C++'s CRT memory tracing functionality, and have found a neat little trick for doing so, using STLSoft's scoped_handle, as follows:

#if defined(_DEBUG) && \
defined(STLSOFT_COMPILER_IS_MSVC)
    int prev = _CrtSetDbgFlag(_CRTDBG_REPORT_FLAG);
    _CrtSetDbgFlag(prev & ~_CRTDBG_ALLOC_MEM_DF);
    stlsoft::scoped_handle scoper(prev, _CrtSetDbgFlag);
#endif

    . . . // allocate memory that wont' be recorded as leaks

} // MSVCRT settings reset here

Wide String Shims for std::exception

2010-11-08T13:02:00.000-08:00

The latest release of STLSoft (1.9.102) includes wide string additions to the string access shims for std::exception (and derived types). The most obvious and immediate benefit is that code using FastFormat and/or Pantheios in wide string builds can now work seamlessly with exceptions, just as has always been the case in multibyte string builds:

#include <fastformat/ff.hpp>

. . . 

#include <pantheios/pan.hpp>

. . . 

int main()

{

. . . 

  catch(std::exception& x)

{

    pan::log_CRITICAL(L"exception: ", x); 

    ff::fmtln(std::wcerr, L"program: {0}", x);

}

. . . 

Dealing with "deprecated" std::copy()

2010-06-22T22:23:00.000-07:00

Although the issue of Microsoft having chosen, by the agency of its so-called "safe string" library, to deprecate large parts of the standard library in general, and std::copy() in particular, is hardly new, there's still some confusion in the community about what to do about it, and how to work around it if that's what you choose to do.

If, as I do on occasion, you choose not to suppress the warnings (by ignoring 4996, or by defining _SCL_SECURE_NO_WARNINGS) but instead want to ignore specifically for std::copy(), you can do it as follows:

#include <stlsoft/algorithms/std/alt.hpp>

#include <stlsoft/internal/safestr.h> 

#if defined(STLSOFT_USING_SAFE_STR_FUNCTIONS) && \
defined(STLSOFT_COMPILER_IS_MSVC)
namespace std
{
using ::stlsoft::std_copy;
# define copy std_copy
}
#endif // compiler

By all means it's breaking the rules and introducing names into the std namespace. Nonetheless, we can sleep easy, since, hey, we didn't start the undermining of the standard!

The other problem is that, currently, stlsoft::std_copy() has no specialisations. It always uses long-hand iteration; no memcpy()-based block copying for random access iterators to POD types.

I hope you know me well enough by now for this to not need to be said. But it's important enough for me to have to say it explicitly, just in case: do not use this technique in library headers!

STLSoft 1.9.98 changes to stlsoft::split()

2010-06-06T17:31:00.000-07:00

As of STLSoft 1.9.98, stlsoft::split() has been enhanced to be able to split into between two and six fragments. Previously you would have use intermediates to split into more fields, as in:

std::string line = "abc|def|ghi|jkl";

stlsoft::string_view   dummy1;
stlsoft::string_view   dummy2;

stlsoft::string_view field0;

stlsoft::string_view field1;
stlsoft::string_view field2;
stlsoft::string_view field3;

if(stlsoft::split(line, '|', field0, dummy1) &&
   stlsoft::split(dummy1, '|', field1, dummy2) &&
   stlsoft::split(dummy2, '|', field2, field3))
{
  . . . // use fields
} 

Although there's no additional memory allocation here - because we're using string views as the intermediate and final fragment types - it's still hard to follow, and doing three separate split operations.

You can now split directly into up to six fields, as in:

std::string  line = "abc|def|ghi|jkl";

stlsoft::string_view field0;

stlsoft::string_view  field1;
stlsoft::string_view field2;
stlsoft::string_view  field3;

if(stlsoft::split(line, '|', field0, field1, field2,  field3))
{
  . . . // use fields
} 

It's marginally more efficient when using string views, and substantially more efficient when using string value types (such as std::string and stlsoft::simple_string). In either cases, it's considerably more transparent.

STLSoft 1.9.98 changes to stlsoft::read_line()

2010-06-06T17:20:00.000-07:00

As of STLSoft 1.9.98, stlsoft::read_line() can now read from an iterator range of arbitrary type. Where formerly it would only work with a C Stream (i.e. FILE*), as in:

std::string line;

while(stlsoft::read_line(stdin, line

     , stlsoft::read_line_flags::recogniseAll))

{

  . . . // process line 

}

Now it will also work with an iterator range of arbitrary type, as in:

platformstl::memory_mapped_file f("file.txt");

char const* const begin = static_cast(f.memory());

char const* const end = begin + f.size();

std::string  line;

while(stlsoft::read_line(begin, end,  line

     ,  stlsoft::read_line_flags::recogniseAll))

{

  . . . // process line 

}

STLSoft 1.9.98 released

2010-06-06T17:03:00.000-07:00

STLSoft 1.9.98 is released, containing a bunch of enhancements for 64-bit compatibility, and in anticipation of STLSoft 1.10. The two particular additions of interest are:

stlsoft::split() can now split into more than two string fragments
stlsoft::read_line() can now work with an iterator range of arbitrary type

STLSoft 1.9.96: Visual C++ support

2010-03-09T15:28:00.000-08:00

The latest STLSoft now supports Visual C++ 10.

VC++ 10 support imminent

2010-03-08T20:15:00.000-08:00

Just added VC++ 10 support to STLSoft, and am proving it with builds of various dependent libraries, including FastFormat, Pantheios and recls. Should be available very soon (1-2 days).

STLSoft 1.9.94: defective definition of swap() overloads for unixstl/winstl::memory_mapped_file corrected

2010-03-04T23:43:00.000-08:00

As of version 1.9.94, the swap() free function overloads for unixstl::memory_mapped_file and winstl::memory_mapped_file are now declared inline, correcting a link-time error when linking multiple compilation units that use the component(s).

STLSoft 1.9.93: STLSOFT_CF_NAMESPACE_SUPPORT now only in C++ compilation units

2010-02-15T23:39:00.000-08:00

As of version 1.9.93, the symbol STLSOFT_CF_NAMESPACE_SUPPORT is now defined only in C++ compilation units. Its (former) definition in C compilation units was meaningless, and put client code to more work, discrimination for STLSOFT_CF_NAMESPACE_SUPPORT and for __cplusplus. Now the former encapsulates both.

STLSoft 1.9.92: inetstl::filesystem_traits::is_path_name_separator() added

2010-02-11T23:27:00.000-08:00

As of version 1.9.92, inetstl::filesystem_traits now has an is_path_name_separator(), to ensure correct handle of paths that contain network ('/') and local ('\\') path name separators.

STLSoft 1.9.91: platformstl::remove_directory_recurse()

2010-02-02T23:27:00.000-08:00

As of version 1.9.91, the UNIXSTL/WinSTL directory function remove_directory_recurse() is introduced into the platformstl namespace.

STLSoft 1.9.90: unixstl::filesystem_traits::get_full_path_name() defect fixed

2010-01-21T23:21:00.000-08:00

Prior to version 1.9.90, unixstl::filesystem_traits::get_full_path_name() contained a defect, causing it to fault if passed "." - indicating the local directory - and instead contain spurious information. This is now fixed.

STLSoft 1.9.88: fixed conversion between const_reverse_iterator and reverse_iterator

2010-01-12T23:50:00.000-08:00

For a long while, I've been aware of the limitations of the reverse iterator abstraction components in the STLSoft libraries - specifically and - which prevent assigning an instance of a collection's reverse_iterator to an instance of its const_reverse_iterator. This has caused trouble for me on occasion, and also for some users.

As of version 1.9.88 this has been addressed, via addition of conversion constructor template and operator != templates.

STLSoft 1.9.88: throwOnAccessFailure search flag for winstl::basic_findfile_sequence

2010-01-12T23:30:00.000-08:00

As anticipated, winstl::basic_findfile_sequence, which is described in detail in Extended STL, volume 1, now defines, and responds to, the member constant throwOnAccessFailure. In the case of an access failure an instance of winstl::access_exception is thrown.

This allows for better response in cases where access may be denied.

STLSoft 1.9.88: max_size() for file_path_buffer and path

2010-01-12T22:40:00.000-08:00

As of version 1.9.88, winstl::basic_file_path_buffer now has a (static) max_size() method, and winstl::basic_path::max_size() is changed to be static.

STLSoft 1.9.88: all-integer handling for comstl::variant

2010-01-12T03:35:00.000-08:00

The latest release (1.9.88) of STLSoft contains an updated comstl::variant class. The previous version had three constructors for integer types: short, int and long. The new version has six constructors for integer types: sint8_t, uint8_t, sint16_t, uint16_t, sint32_t, uint32_t. (It also has short, int and long constructors for those compilers that have distinct types against the 8/16/32-fixed integer types; see chapter 29 of Imperfect C++)

STLSoft 1.9.88 released

2010-01-11T18:51:00.000-08:00

STLSoft 1.9.88 is released today.

This version will be required for forthcoming releases, including:

STLSoft 1.9.88 imminent

2009-11-21T20:47:00.000-08:00

I'm just preparing a new release of recls, and in the process have needed to enhance winstl::basic_findfile_sequence, adding a new throwOnAccessFailure flag, which (as should be obvious from the name), causes an exception to be thrown in the event of access failure. This will allow consistent behaviour between recls (C/C++) and the new recls 100% .NET library.

frequency_map: new merge() method

2009-05-29T19:28:00.000-07:00

The stlsoft::frequency_map container class template now includes a merge() method, which allows the contents from two frequency map instances to be merged into one, as in:

typedef std::string string_t;
typedef stlsoft::frequency_map<string_t> fmap_t;

fmap_t map1;
fmap_t map2;

map1.push("key-11");
map1.push("key-22");

XTESTS_TEST_BOOLEAN_FALSE(map1.empty());
XTESTS_TEST_INTEGER_EQUAL(2u, map1.size());
XTESTS_TEST_INTEGER_EQUAL(1u, map1["key-11"]);
XTESTS_TEST_INTEGER_EQUAL(1u, map1["key-22"]);

map2.push("key-11");
map2.push("key-21");

XTESTS_TEST_BOOLEAN_FALSE(map2.empty());
XTESTS_TEST_INTEGER_EQUAL(2u, map2.size());
XTESTS_TEST_INTEGER_EQUAL(1u, map2["key-11"]);
XTESTS_TEST_INTEGER_EQUAL(1u, map2["key-21"]);

map1.merge(map2);

XTESTS_TEST_BOOLEAN_FALSE(map1.empty());
XTESTS_TEST_INTEGER_EQUAL(3u, map1.size());
XTESTS_TEST_INTEGER_EQUAL(2u, map1["key-11"]);
XTESTS_TEST_INTEGER_EQUAL(1u, map1["key-21"]);
XTESTS_TEST_INTEGER_EQUAL(1u, map1["key-22"]);

XTESTS_TEST_BOOLEAN_FALSE(map2.empty());
XTESTS_TEST_INTEGER_EQUAL(2u, map2.size());
XTESTS_TEST_INTEGER_EQUAL(1u, map2["key-11"]);
XTESTS_TEST_INTEGER_EQUAL(1u, map2["key-21"]);

map2.clear();

XTESTS_TEST_BOOLEAN_FALSE(map1.empty());
XTESTS_TEST_INTEGER_EQUAL(3u, map1.size());
XTESTS_TEST_INTEGER_EQUAL(2u, map1["key-11"]);
XTESTS_TEST_INTEGER_EQUAL(1u, map1["key-21"]);
XTESTS_TEST_INTEGER_EQUAL(1u, map1["key-22"]);

XTESTS_TEST_INTEGER_EQUAL(0u, map2.size());
XTESTS_TEST_BOOLEAN_TRUE(map2.empty());

STLSoft 1.10 new additions: integer_to_array()

2009-05-23T14:23:00.000-07:00

A new component with STLSoft 1.10 (alpha 11 onwards) is the stlsoft::integer_to_array() function template. It is used to turn a single integer value into an array of bit-chunk values.

For example, given the integer i with value 0x01020304, we can split this into the 8 nibbles as follows:

stlsoft::integer_array r = stlsoft::integer_to_array(i, 4);

r[0] will be 4, r[1] will be 0, r[2] will be 3, r[3] will be 0, r[4] will be 2, r[5] will be 0, r[6] will be 1, and r[7] will be 0.

The split can be on any value between 0 and the bitsize of the input parameter (which can be any of the integral types); so you can split on 3 bits, 1 bit, 17 bits, whatever.

I've done extensive automated tests, and they all pass (natch), but I'm still a little dubious about the component, so I'm definitely interested in feedback.

In case you're wondering, the original rationale for this was a simple way to go from the s_addr member of struct in_addr, which is held in network byte order but is otherwise "opaque".

LP64 and -Wshorten-64-to-32

2009-05-23T14:12:00.000-07:00

Along with new Mac OS-X 64-bit makefiles for FastFormat and Pantheios, I'm also playing around with the -Wshorten-64-to-32 warning flag.

In compiling FF with this I encountered a lot of similars, amounting to the following:

enum { x = sizeof(Y) };

Obviously an enum, being int in size, is too small to hold size_t (the result of sizeof operator). The ugly but effective solution to this is:

enum { x = int(sizeof(Y)) };

which you'll now see more of in the STLSoft libs.

Safely abstracting use of strerror / strerror_s (part 1)

2009-05-22T20:15:00.000-07:00

A recent bug report on the STLSoft SourceForge project site reported that the stlsoft::error_desc component did not use the new "safe string" library function strerror_s(). This was actually a surprise, because I thought I'd already taken care of that.

Since I hadn't, I decided that I should. The change in implementation to use strerror_s() (when in the presence of the "safe string" library) goes along the lines of the following


char buff[1001];
if(0 != ::strerror_s(buff, 
            STLSOFT_NUM_ELEMENTS(buff) - 1, errno))
{
  buff[0] = '\0';
}
else
{
  buff[STLSOFT_NUM_ELEMENTS(buff) - 1] = '\0';
}

The dumb part of the strerror_s() function is that it doesn't tell you how many characters were received. This means that you cannot rely on having elicited the full message unless ::strlen() over the returned string is less than (buffer size - 1). So, the above code could return a partial error string (although the likelihood of that is, of course, vanishingly small).

Instead, what I've done is used an auto_buffer to provide resizable storage, and then strerror_s() is called in a loop until either it fails, or no more storage can be allocated, or the length of the returned string is less than (buffer size - 1). The code looks like the following:


stlsoft::auto_buffer buff(128);

for(;;)
{
  int n = ::strerror_s(&buff[0], buff.size() - 1, error);

  buff[buff.size() - 1u] = '\0';

  if(0 == n)
  {
    size_t cch = ::strlen(buff.data());

    if(cch < buff.size() - 2u)
    {
      m_length = cch;
      buff.resize(cch + 1u);
      break;
    }
  }

  if(!buff.resize(1u + buff.size() * 2u))
  {
    buff.resize(1u);
    break;
  }
}

There's another problem with the class template, but that'll have to wait until a later time to discuss ...

WinSTL Registry library mods and fixes, part 3: exception-safety

2009-05-18T01:49:00.001-07:00

In reviewing the implementation of the WinSTL Registry Library's winstl::basic_reg_value class - as described in part 1 and part 2 of this series of posts - I've also spotted a defect in exception-safety.

Consider the (chopped-down) definition of the basic_reg_value class:

template < . . . >
class basic_reg_value
{
. . .
private:
hkey_type    m_hkey; // The parent key of the value
string_type  m_name; // The name of the value
. . . // other members
};

The m_hkey member is obtained via winstl::reg_traits<>::reg_dup_key(). It is the basic_reg_value class itself that provides the RAII. Consequently, if any exception occur during its constructor, the release of m_hkey will not be carried out.

Since the m_name is a string class instance, its constructor can throw. Consequently, basic_reg_value is not exception safe.

Thankfully, the fix is very simple. Simply reverse the order of declaration of the two members. If m_name's constructor throws, that'll happen before the key duplication takes place. If the key duplication throws an exception, the (fully constructed) m_name's destructor will be invoked. Q.E.D.

WinSTL Registry library mods and fixes, part 2: race conditions

2009-05-18T01:21:00.000-07:00

As discussed at great length in section 33.3 of Extended STL, volume 1, the Windows Registry API is one that is prone to race conditions, due to the fact that separate processes may make independent changes to the registry contents without any control over each other.

The recently discovered defect in the WinSTL Registry Library's winstl::basic_reg_value class, gave me cause to consider the implementation in detail again. It's been a long time since I've done that, and with the understanding of the registry race-conditions I gained while researching and writing Extended STL, I saw immediately the possibility of such a race accounting for the reported fault.

Consider again the implementation of winstl::basic_reg_value<>::value_sz() method. Assume that prior to the invocation of winstl::reg_traits<>::reg_query_info that the registry-value's value was non-zero size. The call commences. Meanwhile, another process overwrites the registry-value, with a zero size. reg_query_info returns, and indicates that the data size is zero. Without a further check on the data size, the same fault will be experienced. Naturally, the fix for the non-race defect will fix the race one as well. Which is nice.

WinSTL Registry library mods and fixes, part 1: empty values

2009-05-18T00:59:00.001-07:00

An STLSoft user recently posted a possible defect in the implementation of the WinSTL Registry Library's winstl::basic_reg_value class, reporting that a registry value (of type REG_SZ) yields a data value of size 0, leading to a crash.

Upon first examination, I thought this was a result of the fragility of the Windows Registry with respect to race conditions, as I'll discuss in a follow-up post.

However, closer examination reveals it to be a true defect. The precise circumstances in which this occur are as follows:

the registry-value whose value is being elicited - as a string (REG_SZ) or as an array of strings (REG_MULTI_SZ) has zero size, and
it has one or more peer registry-values whose values are of non-zero size

This precise set of circumstances causes the defect to fault. The reason lies in a call to winstl::reg_traits<>::reg_query_info at the start of the winstl::basic_reg_value<>::value_sz() method. This is used to determine the maximum size of the value of any if the current key's registry-values. This is useful to be able to provide a buffer of the appropriate size to the subsequent call to winstl::reg_traits<>::reg_query_value(), which actually retrieves the value in question.

The problem occurs when the value's size is 0. The last block in the method decrements this - to account for the space for the nul-terminator added earlier - and then explicitly sets the nul-terminator. (I actually forget why it does this, but I do recall that it must be done this way.)

Anyway, when the value's size is 0, decrementing it gives a very large number, and so the next statement results in an access-violation. Yuck!

STLSoft 1.9.83 will contain the fix for this, which is simply to test again that the data size is non-0.

More 1.9.82 ...

2009-05-15T14:03:00.000-07:00

It also includes a new method in winstl::reg_traits::reg_delete_tree(), which takes a key handle and a sub-key name, and deletes the sub-key and any/all its descendent keys, as in:

HKEY k = . . .

LONG res = winstl::reg_traits<char>::reg_delete_tree(k, "sub-key");

Use with care, because there's no un-delete!

Catching up with developments ... 1.9.82

2009-05-15T13:55:00.000-07:00

I've been lax in the blogging - too much to do, too little time - but will try and catch up.

The latest release includes a fix to allow winstl::findfile_sequence - a facade over the FindFirstFile/FindNextFile API to be used with std::copy and the IOStreams, as in:

#include <stlsoft/iterators/ostream_iterator.hpp>

#include <winstl/filesystem/findfile_sequence.hpp>

#include <iostream>

#include <stdlib.h>

int main()

{

typedef winstl::findfile_sequence seq_t;

{ // 1. enumerate all contents (except dots dirs)

seq_t entries("*.*");

std::cout << "\n1:\n";

std::copy( entries.begin(), entries.end()