Showing posts with label FastFormat. Show all posts
Showing posts with label FastFormat. Show all posts

Tuesday, December 28, 2010

Scoping MSVCRT memory tracking flags with scoped_handle

Am just working on some diagnostic extras to be provided as a side-project for Pantheios, with a particular focus on main(). In applying them to some of my system tool programs, I'm finding some false positive memory leaks being reported for FastFormat: this is because FastFormat caches parsed format strings, and they're not released until the library is uninitialised.

I want to "hide" these memory allocations from Visual C++'s CRT memory tracing functionality, and have found a neat little trick for doing so, using STLSoft's scoped_handle, as follows:


#if defined(_DEBUG) && \
    defined(STLSOFT_COMPILER_IS_MSVC)
    int prev = _CrtSetDbgFlag(_CRTDBG_REPORT_FLAG);
    _CrtSetDbgFlag(prev & ~_CRTDBG_ALLOC_MEM_DF);
    stlsoft::scoped_handle scoper(prev, _CrtSetDbgFlag);
#endif

    . . . // allocate memory that wont' be recorded as leaks


  } // MSVCRT settings reset here

Monday, November 8, 2010

Wide String Shims for std::exception

The latest release of STLSoft (1.9.102) includes wide string additions to the string access shims for std::exception (and derived types). The most obvious and immediate benefit is that code using FastFormat and/or Pantheios in wide string builds can now work seamlessly with exceptions, just as has always been the case in multibyte string builds:

#include <fastformat/ff.hpp>
. . .
#include <pantheios/pan.hpp>
. . . 

int main()
{
. . .
  catch(std::exception& x)
  {
    pan::log_CRITICAL(L"exception: ", x);
    ff::fmtln(std::wcerr, L"program: {0}", x);
  }
. . . 

Monday, March 8, 2010

VC++ 10 support imminent

Just added VC++ 10 support to STLSoft, and am proving it with builds of various dependent libraries, including FastFormat, Pantheios and recls. Should be available very soon (1-2 days).

Saturday, May 23, 2009

LP64 and -Wshorten-64-to-32

Along with new Mac OS-X 64-bit makefiles for FastFormat and Pantheios, I'm also playing around with the -Wshorten-64-to-32 warning flag.

In compiling FF with this I encountered a lot of similars, amounting to the following:

enum { x = sizeof(Y) };

Obviously an enum, being int in size, is too small to hold size_t (the result of sizeof operator). The ugly but effective solution to this is:

enum { x = int(sizeof(Y)) };

which you'll now see more of in the STLSoft libs.

Thursday, January 1, 2009

Working with other libraries, part 2: allocators

Much is often made of the supposedly still-born Allocator concept in the standard library. However, one very good use of them is in tracking memory.

At the moment I'm preparing some analyses of FastFormat's performance for an article I'm writing. One of the analyses conducting is to see how many memory allocations are involved in a formatting statement, for each of the comparison libraries. The standard way to achieve something like this is to overload the global operators new, as in:

// NOTE: this code is only valid for single-threaded operation

extern int s_nallocs = 0;

#ifdef OVERLOAD_OPNEW
void* counting_malloc(size_t cb)
{
  ++s_nallocs;

  return ::malloc(cb);
}

void counting_free(void* pv)
{
  ::free(pv);
}

void* operator new(size_t cb)
{
  return counting_malloc(cb);
}

void operator delete(void* pv)
{
  counting_free(pv);
}

void* operator new[](size_t cb)
{
  return counting_malloc(cb);
}

void operator delete[](void* pv)
{
  counting_free(pv);
}
#endif /* OVERLOAD_OPNEW */

Unfortunately, some components with some compilers - the exact permutations escape me at this point - don't go through operator new. This might be because they use a per-class operator new, or it might be because they use an allocator that doesn't use new. (Being under a publishing deadline, I didn't have the time - nor the inclination, if I'm honest - to find out which it was in each case.)

So, in order to get a fighting chance at an accurate depiction of how much memory each library is using I decided to force the issue, by requiring all the strings used to be an instance of the following specialisation, rather than std::string:

typedef std::basic_string<
  char
, std::char_traits
<char>
stlsoft::new_allocator<char>
>   string_t;

Of course, things aren't ever that simple. Such a string type is not compatible with the IOStreams default specialisations, requiring:

typedef std::basic_stringstream<
  char
, std::char_traits
<char>
stlsoft::new_allocator<char>
>   stringstream_t;

And the same thing applies for Boost.Format, requiring:

typedef boost::basic_format<
  char
, std::char_traits
<char>
stlsoft::new_allocator<char>
>   format_t;

Unfortunately, Loki's SafeFormat library does not allow for the specification of allocators (or character traits, for that matter), and only uses std::string. So a little horrifying trickery was required.

Step 1: Introduce string_t into the std namespace.

namespace std
{
  using ::string_t;
}

Now, if you've been paying attention this last decade or so you'll know that adding to the std namespace is strictly controlled. I won't go over the rules now; you can look it up. Suffice to say that this action is not allowed.

Of course, needs must, and in this case there's no choice. Since it's just a perf-test program, it's ok. Just don't go using this tactic in production code.

Step 2: Make Loki (and any other code, for that matter) think that std::string_t is std::string.

#define string string_t

I warned you it was horrid!

Step 3: #include the Loki.SafeFormat header

#include <loki/safeformat.h>

Obviously, this has to be done after steps 1 & 2, otherwise it won't work.


There were a few other dodgy things I had to do to get it to work with some really stupid compilers, but that'll have to wait until another day.

Monday, December 29, 2008

Width/Alignment/Fill functionality for FastFormat (0.3) functionally complete

Work on FastFormat has been progressing substantially over the last few days, in preparation for:
  • A series of articles to feature in Overload in 09
  • Monolith
As part of this, the width/alignment/fill functionality has been proven in the (currently unreleased) 0.3 branch, so that minimum-width and/or maximum-width and/or alignment and/or fill may be specified. Let's consider a couple of examples.

1. Output a string that uses a single argument twice, with default formatting in each case:

ff::fmtln(std::cout, "x={0}; y={0};", "abc");

This prints x=abc; y=abc;


2. Specify a minimum width for the first parameter:

ff::fmtln(std::cout, "x={0}; y={0,10};", "abc");

This prints x=abc; y=       abc;


3. Specify a minimum width and right alignment (which is the default) for the first parameter:

ff::fmtln(std::cout, "x={0}; y={0,10,,>};", "abc");

This prints x=abc; y=       abc;



4. Specify a minimum width and left alignment for the first parameter:

ff::fmtln(std::cout, "x={0}; y={0,10,,<};", "abc");

This prints x=abc; y=abc       ;


5. Specify a minimum width and centre alignment for the first parameter:

ff::fmtln(std::cout, "x={0}; y={0,10,,^};", "abc");

This prints x=abc; y=   abc    ;


6. Specify a maximum width for the second parameter:

ff::fmtln(std::cout, "x={0}; y={0,,2};", "abc");

This prints x=abc; y=bc;


7. Specify a maximum width and left alignment for the second parameter:

ff::fmtln(std::cout, "x={0}; y={0,,2,<};", "abc");

This prints x=abc; y=ab;


The precise semantics of the alignment and fill (not shown above) options may still be changed, but the basis of the parameter syntax (four fields: index, min-width, max-width, alignment-and-fill) seem to work out well.

And the really great news late last night was that this new functionality has no appreciable cost over 0.2, so FastFormat is still head and shoulders ahead of Boost.Format, Loki.SafeFormat, and the IOStreams in performance (as it also is in robustness, expressiveness and flexibility).

I've released beta 7 of 0.2.1 today, and hope to be releasing an alpha of 0.3 in the next couple of weeks.

FastFormat now performance tested against Loki

As of 0.2.1 beta 6, FastFormat's performance test programs now also compare against Andrei Alexandrescu's Loki library's SafeFormat component. The results clearly demonstrate FastFormat's performance superiority over this library, as they do over C++'s standard IOStreams and Boost.Format.

Given the fact that FastFormat is more robust and more flexible than these other libraries, and is highly expressive, I think it's fair to now claim that it is the pre-eminent formatting library for C++. All that remains is to provide the planned width+alignment+fill functionality, and it'll be effectively complete.

I'll be looking for input/assistance in the new year for packaging and porting. If anyone wants to volunteer, you'll be most welcome.

Friday, December 12, 2008

Radio silence ...

Having had a protracted break from STLSoft-related blogging, I'm back.

I've been busy with clients, and with various coding and writing projects, of which more in the coming posts.

I'll be attempting to post daily for the next few weeks, as I make my way through the following tasks:

Friday, November 21, 2008

STLSoft 1.10 new additions: string_to_integer

Having not posted for over a month, there's a lot to catch up on.

I'm going to start with a series of short missives on the latest additions to STLSoft 1.10. (See this for how to install/use 1.10 alpha (delta) releases.) While I'm catching up with the releases, it's probably not going to go in order, so you'll have to be bear with me.

I'll start today with the new stlsoft::string_to_integer() functions, defined in the new include file stlsoft/conversion/string_to_integer.hpp. These are low-level string->number parsing functions. Their raison d'etre is performance - they'll be used in the forthcoming FastFormat 0.3 version, for high-speed parsing of replacement parameters - and for this they sacrifice some functionality, as discussed below.

They look like the following:

namespace stlsoft
{
int string_to_integer(char const* s, char const** endptr);

int string_to_integer(wchar_t const* s, wchar_t const** endptr);
int string_to_integer(char const* s, size_t len, char const** endptr);
int string_to_integer(wchar_t const* s, size_t len, wchar_t const** endptr);
} // namespace stlsoft


The functions in the first pair take a (non-NULL) pointer to a (nul-terminated) C-style string, along with an optional (may be NULL) pointer to pointer to C-style string, which will receive the pointer to the character terminating the parsing. In this aspect, these functions ape the functionality of the standard strtol() function. However, where they differ is that they do not accept a radix parameter: currently all conversion is decimal.

The second pair do not require nul-terminated strings, and instead use a length parameter. This is useful when parsing numbers out of string types that may not be null-terminated. (Again, this is found in FastFormat, which is the main driver for the release of these functions.)

These functions are very new, and will evolve further before STLSoft 1.10 beta. Three changes that are candidates to be introduced are:
  • Supporting different radixes (although for radixes not in [8, 10, 16] it may be that they'd defer to strtol()).
  • Support for different integer types. In fact, the four concrete functions are implemented in terms of two function templates whose integral type is a template parameter, so the groundwork is all there
  • Support for truncation testing.
I'd like to hear from anyone on these, or other, features.

I'll do some specific performance tests on the functions at a later time. They've had some exposure as part of a larger performance test of the forthcoming FastFormat 0.3's new functionality, and have shown to confer a benefit over strtol() and its siblings.

Tuesday, October 14, 2008

Release early, release often; FastFormat 0.2.1 (alpha 4)

The value of the open-source adage to "release early, release often" has shown itself once again, this time with respect to the FastFormat library. There are a large number of things to go into this library on its progress from 0.2 onwards, but it's been stuck for a month waiting for me to get my act together. Part of the delay is due to the fact that I let too many changes creep in, which complicates the testing and the formulation of the distribution (incl. just listing the changes).

So, I've just released 0.2.1 (alpha 4), which is a sound base for the future releases. It incorporates various minor fixes and adjustments, and a wholesale refactoring of the contract enforcement API. (An analogous refactoring has also been done for Pantheios, and is available in its latest release.)

Wednesday, October 8, 2008

Busy times; Pantheios.COM release

Things are really busy at the moment, what with my current commercial client and doing a fair chunk of the writing commitments that I've been dodging for much of this year.

I'm committed to writing 8 articles over the next year, about things to do with Monolith (FastFormat, Pantheios, Type-tunneling, Shims, and so on).

But in the meantime things are progressing on the libraries. Today I've released a new version of Pantheios.COM, which fixes incompatibility with early-binding automation clients (such as Visual Basic and Delphi).

Friday, September 26, 2008

New Libraries coming ...

Commercial and publishing reasons are forcing me to do what I should have done a *long* time ago, and release several pending libraries in the coming days:
* xContract - a contract programming enforcement library; initially for C/C++
* xCover - a code coverage library for C and C++
* FastFormat - already available as an alpha, this will hit the non-alpha/non-beta status in the coming days; as the name implies, it's faster than all the competition put together ;-)
* Pantheios - in beta (and very popular) for two years, the world's fastest and most robust logging API library has waited long enough for a final 1.0 release.

Also, STLSoft is going to be moving to SourceForge in the next week or so, and it too will receive a face lift and an update to docs, tests, and so on.

If anyone wants to volunteer any help for any of these activities, you'd be extraordinarily welcome.

Cheers

Matt

Thursday, September 11, 2008

FastFormat 0.2.1 (alpha 3) released

FastFormat 0.2.1 (alpha 3) is released.

It contains a fix to a defect in the implicit link header, and also a lot of groundwork for widestring support in the near future. (Actually, all the widestring support is now in the core library - just the makefiles, examples and tests need to be correspondingly updated.)

Wednesday, September 10, 2008

FastFormat is FAST: It's official!

I just posted some performance stats on the FastFormat website. I'll just give you the summary here:
  • The Format API is faster than IOStreams by between ~140-730%, faster than MFC's CString::Format() by between ~300-400%, and faster than Boost.Format by between ~470-1600%! The only formatting API that gives it a run for some architecture/compiler/configurations is sprintf(), which is not type-safe, at between ~40-380%. (FastFormat's Format API is 100% type-safe.)
  • The simpler Write API is faster than IOStreams by between ~270-1350%, faster than MFC's CString::Format() by around ~420%, and faster than Boost.Format by between ~630-1800%! Again, the only formatting API that gives it a run for some architecture/compiler/configurations is sprintf(), at between ~65-390%. (FastFormat's Write API is 100% type-safe.)

Wednesday, September 3, 2008

FastFormat is unleashed: v0.2.1 (alpha 1) released

FastFormat is an Open Source C/C++ Output/Formatting library, whose design parameters are 100% type-safety, efficiency, genericity and extensibility. It is simple to use and extend, highly-portable (platform and compiler-independent) and, best of all, it upholds the C tradition of you only pay for what you use.

FastFormat supports output/formatting of statements of arbitrary complexity, consisting of heterogeneous types.

FastFormat writes to output "sinks", which can be of arbitrary type. It implicitly supports any type that is structurally conformant with the standard library's string, and the library includes adaptors to allow writing to std::ostream, FILE*, speech (currently Windows-only), STLSoft's auto_buffer, C-style string buffers, and character buffers. Adaptation to a new type merely requires the definition of a single function.

FastFormat is fast. The processing of each statement involves at most one memory allocation to hold the entire statement, and each statement element is measured and copied exactly once. As a consequence, the library is on a par with (the type-unsafe) C's Streams (printf()-family) of functions, faster than C++'s IOStreams, and considerably faster than Boost.Format. Comprehensive performance analyses are underway; initial results of a realistic scenario on Windows (32-bit) with Visual C++ 9 shows FastFormat is approximately 3 x faster than C's Streams, 6 x faster than C++'s IOStreams, and 17 x faster than Boost.Format.

FastFormat supports I18N/L10N by using numbered arguments, enabling reordering of arguments by exchanging format strings. The library comes with a number of resource bundles, classes whose instances can load sets of localised resource strings for use as format strings.

FastFormat does not contain any compiler-specific or platform-specific constructs. It supports UNIX (including Linux and Mac OS-X), and Windows, and should work with any operating system. It is known to be compatible with Comeau (4.3.3+), GCC (3.4+), Intel (8+), Metrowerks (8+), Microsoft Visual C++ (6.0+), and should work with any reasonably modern C++ compiler.

FastFormat is completely free and includes source released under a BSD-style license. Commercial customisations and related consultancy are provided by Synesis Software Pty Ltd; http://synesis.com.au/contact.html)

Release 0.2.1 alpha 1 is the first public release.

Note: this release of FastFormat requires STLSoft 1.9.50, or later. Download from http://stlsoft.org/

Download from: http://sourceforge.net/project/showfiles.php?group_id=177382&package_id=204396

Discuss at: http://sourceforge.net/forum/forum.php?forum_id=612781

FastFormat website: http://fastformat.org/

Monday, September 1, 2008

FastFormat about to drop ... and it's FAST!

I'm now less than two days away from releasing FastFormat. All that's left to do is:
  • verify on 64-bit Linux
  • do the docs (a basic first cut; nothing mega good)
  • try and find out why DMC++ does not like it (as usual, it's the odd-man-out with templates)
  • do a decently comprehensive lot of performance tests
As a first go at verifying the performance, I've run a test which formats thusly:

std::string arg0 = "abc";

const char arg1[] = "def";

stlsoft::simple_string arg2 = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";


fastformat::fmt(s, "The first param '{0}', and the second '{1}', and the first again '{0}'. Finally the third '{2}'", arg0, arg1, arg2);


Note: this is for one compiler (VC++ 9), and one scenario (described above), so we can't draw any way-out conclusions as yet. But the first results are pretty encouraging to say the least:

IOStreams: 679561us
Boost.Format: 1901109us
sprintf(): 346776us
FastFormat: 112016us
IOStreams:FastFormat: 6.067
Boost.Format:FastFormat: 16.97
sprintf:FastFormat(): 3.096

These numbers mean that, for that scenario with that compiler, FastFormat is 3x faster than sprintf, 6x faster than the IOStreams, and 17x than Boost.Format.

Given that it's 100% type-safe, infinitely extensible, and supports I18N/L10N, I hope that it's going to be well received.

Thursday, August 28, 2008

... FastFormat?

For those long-suffering waitees for the long-promised revolution in string formatting that is FastFormat, I promise that the wait won't be much longer. I've got a few things on my plate over the next few days, and then FF is top of the list. All that's left to do is sort out a few distro issues, and tidy up the docs, and we're ready to rock and roll. (At least for an alpha version, anyway.)