mame/src/lib/util/flac.c
Aaron Giles f0823886a6 Major CHD/chdman update. The CHD version number has been increased
from 4 to 5. This means any diff CHDs will no longer work. If you
absolutely need to keep the data for any existing ones you have,
find both the diff CHD and the original CHD for the game in question 
and upgrade using these commands:

  rename diff\game.dif diff\game-old.dif
  chdman copy -i diff\game-old.dif -ip roms\game.chd -o diff\game.dif -op roms\game.chd -c none

Specifics regarding this change:

Defined a new CHD version 5. New features/behaviors of this version:
  - support for up to 4 codecs; each block can use 1 of the 4
  - new LZMA codec, which tends to do better than zlib overall
  - new FLAC codec, primarily used for CDs (but can be applied anywhere)
  - upgraded AVHuff codec now uses FLAC for encoding audio
  - new Huffman codec, used to catch more nearly-uncompressable blocks
  - compressed CHDs now use a compressed map for significant savings
  - CHDs now are aware of a "unit" size; each hunk holds 1 or more units
     (in general units map to sectors for hard disks/CDs)
  - diff'ing against a parent now diffs at the unit level, greatly
     improving compression

Rewrote and modernized chd.c. CHD versions prior to 3 are unsupported,
and version 3/4 CHDs are only supported for reading. Creating a new
CHD now leaves the file open.  Added methods to read and write at the 
unit and byte level, removing the need to handle this manually. Added
metadata access methods that pass astrings and dynamic_buffers to
simplify the interfaces. A companion class chd_compressor now
implements full multithreaded compression, analyzing and compressing
multiple hunks independently in parallel. Split the codec 
implementations out into a separate file chdcodec.*

Updated harddisk.c and cdrom.c to rely on the caching/byte-level read/
write capabilities of the chd_file class. cdrom.c (and chdman) now also 
pad CDs to 4-frame boundaries instead of hunk boundaries, ensuring that
the same SHA1 hashes are produced regardless of the hunk size.

Rewrote chdman.exe entirely, switching from positional parameters to
proper options. Use "chdman help" to get a list of commands, and
"chdman help <command>" to get help for any particular command. Many 
redundant commands were removed now that additional flexibility is
available. Some basic mappings:

  Old: chdman -createblankhd <out.chd> <cyls> <heads> <secs>
  New: chdman createhd -o <out.chd> -chs <cyls>,<heads>,<secs>

  Old: chdman -createuncomphd <in.raw> <out.chd> ....
  New: chdman createhd -i <in.raw> -o <out.chd> -c none ....

  Old: chdman -verifyfix <in.chd>
  New: chdman verify -i <in.chd> -f

  Old: chdman -merge <parent.chd> <diff.chd> <out.chd>
  New: chdman copy -i <diff.chd> -ip <parent.chd> -o <out.chd>

  Old: chdman -diff <parent.chd> <compare.chd> <diff.chd>
  New: chdman copy -i <compare.chd> -o <diff.chd> -op <parent.chd>

  Old: chdman -update <in.chd> <out.chd>
  New: chdman copy -i <in.chd> -o <out.chd>

Added new core file coretmpl.h to hold core template classes. For now
just one class, dynamic_array<> is defined, which acts like an array
of a given object but which can be appended to and/or resized. Also
defines dynamic_buffer as dynamic_array<UINT8> for holding an 
arbitrary buffer of bytes. Expect to see these used a lot.

Added new core helper hashing.c/.h which defines classes for each of
the common hashing methods and creator classes to wrap the 
computation of these hashes. A future work item is to reimplement
the core emulator hashing code using these.

Split bit buffer helpers out into C++ classes and into their own
public header in bitstream.h.

Updated huffman.c/.h to C++, and changed the interface to make it
more flexible to use in nonstandard ways. Also added huffman compression
of the static tree for slightly better compression rates.

Created flac.c/.h as simplified C++ wrappers around the FLAC interface.
A future work item is to convert the samples sound device to a modern
device and leverage this for reading FLAC files.

Renamed avcomp.* to avhuff.*, updated to C++, and added support for
FLAC as the audio encoding mechanism. The old huffman audio is still
supported for decode only.

Added a variant of core_fload that loads to a dynamic_buffer.

Tweaked winwork.c a bit to not limit the maximum number of processors
unless the work queue was created with the WORK_QUEUE_FLAG_HIGH_FREQ
option. Further adjustments here are likely going to be necessary.

Fixed bug in aviio.c which caused errors when reading some AVI files.
2012-02-16 09:47:18 +00:00

599 lines
19 KiB
C

/***************************************************************************
flac.c
FLAC compression wrappers
****************************************************************************
Copyright Aaron Giles
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
* Neither the name 'MAME' nor the names of its contributors may be
used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY AARON GILES ''AS IS'' AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL AARON GILES BE LIABLE FOR ANY DIRECT,
INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
***************************************************************************/
#include "flac.h"
#include <assert.h>
#include <new>
//**************************************************************************
// FLAC ENCODER
//**************************************************************************
//-------------------------------------------------
// flac_encoder - constructors
//-------------------------------------------------
flac_encoder::flac_encoder()
{
init_common();
}
flac_encoder::flac_encoder(void *buffer, UINT32 buflength)
{
init_common();
reset(buffer, buflength);
}
flac_encoder::flac_encoder(core_file &file)
{
init_common();
reset(file);
}
//-------------------------------------------------
// ~flac_encoder - destructor
//-------------------------------------------------
flac_encoder::~flac_encoder()
{
// delete the encoder
FLAC__stream_encoder_delete(m_encoder);
}
//-------------------------------------------------
// reset - reset state with the original
// parameters
//-------------------------------------------------
bool flac_encoder::reset()
{
// configure the output
m_compressed_offset = 0;
m_ignore_bytes = m_strip_metadata ? 4 : 0;
m_found_audio = !m_strip_metadata;
// configure the encoder in a standard way
// note we do this on each reset; if we don't, results are NOT consistent!
FLAC__stream_encoder_set_verify(m_encoder, false);
// FLAC__stream_encoder_set_do_md5(m_encoder, false);
FLAC__stream_encoder_set_compression_level(m_encoder, 8);
FLAC__stream_encoder_set_channels(m_encoder, m_channels);
FLAC__stream_encoder_set_bits_per_sample(m_encoder, 16);
FLAC__stream_encoder_set_sample_rate(m_encoder, m_sample_rate);
FLAC__stream_encoder_set_total_samples_estimate(m_encoder, 0);
FLAC__stream_encoder_set_streamable_subset(m_encoder, false);
FLAC__stream_encoder_set_blocksize(m_encoder, m_block_size);
// re-start processing
return (FLAC__stream_encoder_init_stream(m_encoder, write_callback_static, NULL, NULL, NULL, this) == FLAC__STREAM_ENCODER_INIT_STATUS_OK);
}
//-------------------------------------------------
// reset - reset state with new memory parameters
//-------------------------------------------------
bool flac_encoder::reset(void *buffer, UINT32 buflength)
{
// configure the output
m_compressed_start = reinterpret_cast<FLAC__byte *>(buffer);
m_compressed_length = buflength;
m_file = NULL;
return reset();
}
//-------------------------------------------------
// reset - reset state with new file parameters
//-------------------------------------------------
bool flac_encoder::reset(core_file &file)
{
// configure the output
m_compressed_start = NULL;
m_compressed_length = 0;
m_file = &file;
return reset();
}
//-------------------------------------------------
// encode_interleaved - encode a buffer with
// interleaved samples
//-------------------------------------------------
bool flac_encoder::encode_interleaved(const INT16 *samples, UINT32 samples_per_channel, bool swap_endian)
{
int shift = swap_endian ? 8 : 0;
// loop over source samples
int num_channels = FLAC__stream_encoder_get_channels(m_encoder);
UINT32 srcindex = 0;
while (samples_per_channel != 0)
{
// process in batches of 2k samples
FLAC__int32 converted_buffer[2048];
FLAC__int32 *dest = converted_buffer;
UINT32 cur_samples = MIN(ARRAY_LENGTH(converted_buffer) / num_channels, samples_per_channel);
// convert a buffers' worth
for (UINT32 sampnum = 0; sampnum < cur_samples; sampnum++)
for (int channel = 0; channel < num_channels; channel++, srcindex++)
*dest++ = INT16((UINT16(samples[srcindex]) << shift) | (UINT16(samples[srcindex]) >> shift));
// process this batch
if (!FLAC__stream_encoder_process_interleaved(m_encoder, converted_buffer, cur_samples))
return false;
samples_per_channel -= cur_samples;
}
return true;
}
//-------------------------------------------------
// encode - encode a buffer with individual
// sample streams
//-------------------------------------------------
bool flac_encoder::encode(INT16 *const *samples, UINT32 samples_per_channel, bool swap_endian)
{
int shift = swap_endian ? 8 : 0;
// loop over source samples
int num_channels = FLAC__stream_encoder_get_channels(m_encoder);
UINT32 srcindex = 0;
while (samples_per_channel != 0)
{
// process in batches of 2k samples
FLAC__int32 converted_buffer[2048];
FLAC__int32 *dest = converted_buffer;
UINT32 cur_samples = MIN(ARRAY_LENGTH(converted_buffer) / num_channels, samples_per_channel);
// convert a buffers' worth
for (UINT32 sampnum = 0; sampnum < cur_samples; sampnum++, srcindex++)
for (int channel = 0; channel < num_channels; channel++)
*dest++ = INT16((UINT16(samples[channel][srcindex]) << shift) | (UINT16(samples[channel][srcindex]) >> shift));
// process this batch
if (!FLAC__stream_encoder_process_interleaved(m_encoder, converted_buffer, cur_samples))
return false;
samples_per_channel -= cur_samples;
}
return true;
}
//-------------------------------------------------
// finish - complete encoding and flush the
// stream
//-------------------------------------------------
UINT32 flac_encoder::finish()
{
// process the data and return the amount written
FLAC__stream_encoder_finish(m_encoder);
return (m_file != NULL) ? core_ftell(m_file) : m_compressed_offset;
}
//-------------------------------------------------
// init_common - common initialization
//-------------------------------------------------
void flac_encoder::init_common()
{
// allocate the encoder
m_encoder = FLAC__stream_encoder_new();
if (m_encoder == NULL)
throw std::bad_alloc();
// initialize default state
m_file = NULL;
m_compressed_offset = 0;
m_compressed_start = NULL;
m_compressed_length = 0;
m_sample_rate = 44100;
m_channels = 2;
m_block_size = 0;
m_strip_metadata = false;
m_ignore_bytes = 0;
m_found_audio = false;
}
//-------------------------------------------------
// write_callback - handle writes to the
// output stream
//-------------------------------------------------
FLAC__StreamEncoderWriteStatus flac_encoder::write_callback_static(const FLAC__StreamEncoder *encoder, const FLAC__byte buffer[], size_t bytes, unsigned samples, unsigned current_frame, void *client_data)
{
return reinterpret_cast<flac_encoder *>(client_data)->write_callback(buffer, bytes, samples, current_frame);
}
FLAC__StreamEncoderWriteStatus flac_encoder::write_callback(const FLAC__byte buffer[], size_t bytes, unsigned samples, unsigned current_frame)
{
// loop over output data
size_t offset = 0;
while (offset < bytes)
{
// if we're ignoring, continue to do so
if (m_ignore_bytes != 0)
{
int ignore = MIN(bytes - offset, m_ignore_bytes);
offset += ignore;
m_ignore_bytes -= ignore;
}
// if we haven't hit the end of metadata, process a new piece
else if (!m_found_audio)
{
assert(bytes - offset >= 4);
m_found_audio = ((buffer[offset] & 0x80) != 0);
m_ignore_bytes = (buffer[offset + 1] << 16) | (buffer[offset + 2] << 8) | buffer[offset + 3];
offset += 4;
}
// otherwise process as audio data and copy to the output
else
{
int count = bytes - offset;
if (m_file != NULL)
core_fwrite(m_file, buffer, count);
else
{
if (m_compressed_offset + count <= m_compressed_length)
memcpy(m_compressed_start + m_compressed_offset, buffer, count);
m_compressed_offset += count;
}
offset += count;
}
}
return FLAC__STREAM_ENCODER_WRITE_STATUS_OK;
}
//**************************************************************************
// FLAC DECODER
//**************************************************************************
//-------------------------------------------------
// flac_decoder - constructor
//-------------------------------------------------
flac_decoder::flac_decoder()
: m_decoder(FLAC__stream_decoder_new()),
m_file(NULL),
m_compressed_offset(0),
m_compressed_start(NULL),
m_compressed_length(0),
m_compressed2_start(NULL),
m_compressed2_length(0)
{
}
//-------------------------------------------------
// flac_decoder - constructor
//-------------------------------------------------
flac_decoder::flac_decoder(const void *buffer, UINT32 length, const void *buffer2, UINT32 length2)
: m_decoder(FLAC__stream_decoder_new()),
m_file(NULL),
m_compressed_offset(0),
m_compressed_start(reinterpret_cast<const FLAC__byte *>(buffer)),
m_compressed_length(length),
m_compressed2_start(reinterpret_cast<const FLAC__byte *>(buffer2)),
m_compressed2_length(length2)
{
reset();
}
//-------------------------------------------------
// flac_decoder - constructor
//-------------------------------------------------
flac_decoder::flac_decoder(core_file &file)
: m_decoder(FLAC__stream_decoder_new()),
m_file(&file),
m_compressed_offset(0),
m_compressed_start(NULL),
m_compressed_length(0),
m_compressed2_start(NULL),
m_compressed2_length(0)
{
reset();
}
//-------------------------------------------------
// flac_decoder - destructor
//-------------------------------------------------
flac_decoder::~flac_decoder()
{
FLAC__stream_decoder_delete(m_decoder);
}
//-------------------------------------------------
// reset - reset state with the original
// parameters
//-------------------------------------------------
bool flac_decoder::reset()
{
m_compressed_offset = 0;
if (FLAC__stream_decoder_init_stream(m_decoder, &flac_decoder::read_callback_static, NULL, NULL, NULL, NULL, &flac_decoder::write_callback_static, NULL, &flac_decoder::error_callback_static, this) != FLAC__STREAM_DECODER_INIT_STATUS_OK)
return false;
return FLAC__stream_decoder_process_until_end_of_metadata(m_decoder);
}
//-------------------------------------------------
// reset - reset state with new memory parameters
//-------------------------------------------------
bool flac_decoder::reset(const void *buffer, UINT32 length, const void *buffer2, UINT32 length2)
{
m_file = NULL;
m_compressed_start = reinterpret_cast<const FLAC__byte *>(buffer);
m_compressed_length = length;
m_compressed2_start = reinterpret_cast<const FLAC__byte *>(buffer2);
m_compressed2_length = length2;
return reset();
}
//-------------------------------------------------
// reset - reset state with new memory parameters
// and a custom-generated header
//-------------------------------------------------
bool flac_decoder::reset(UINT32 sample_rate, UINT8 num_channels, UINT32 block_size, const void *buffer, UINT32 length)
{
// modify the template header with our parameters
static const UINT8 s_header_template[0x2a] =
{
0x66, 0x4C, 0x61, 0x43, // +00: 'fLaC' stream header
0x80, // +04: metadata block type 0 (STREAMINFO),
// flagged as last block
0x00, 0x00, 0x22, // +05: metadata block length = 0x22
0x00, 0x00, // +08: minimum block size
0x00, 0x00, // +0A: maximum block size
0x00, 0x00, 0x00, // +0C: minimum frame size (0 == unknown)
0x00, 0x00, 0x00, // +0F: maximum frame size (0 == unknown)
0x0A, 0xC4, 0x42, 0xF0, 0x00, 0x00, 0x00, 0x00, // +12: sample rate (0x0ac44 == 44100),
// numchannels (2), sample bits (16),
// samples in stream (0 == unknown)
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // +1A: MD5 signature (0 == none)
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 //
// +2A: start of stream data
};
memcpy(m_custom_header, s_header_template, sizeof(s_header_template));
m_custom_header[0x08] = m_custom_header[0x0a] = block_size >> 8;
m_custom_header[0x09] = m_custom_header[0x0b] = block_size & 0xff;
m_custom_header[0x12] = sample_rate >> 12;
m_custom_header[0x13] = sample_rate >> 4;
m_custom_header[0x14] = (sample_rate << 4) | ((num_channels - 1) << 1);
// configure the header ahead of the provided buffer
m_file = NULL;
m_compressed_start = reinterpret_cast<const FLAC__byte *>(m_custom_header);
m_compressed_length = sizeof(m_custom_header);
m_compressed2_start = reinterpret_cast<const FLAC__byte *>(buffer);
m_compressed2_length = length;
return reset();
}
//-------------------------------------------------
// reset - reset state with new file parameter
//-------------------------------------------------
bool flac_decoder::reset(core_file &file)
{
m_file = &file;
m_compressed_start = NULL;
m_compressed_length = 0;
m_compressed2_start = NULL;
m_compressed2_length = 0;
return reset();
}
//-------------------------------------------------
// decode_interleaved - decode to an interleaved
// sound stream
//-------------------------------------------------
bool flac_decoder::decode_interleaved(INT16 *samples, UINT32 num_samples, bool swap_endian)
{
// configure the uncompressed buffer
memset(m_uncompressed_start, 0, sizeof(m_uncompressed_start));
m_uncompressed_start[0] = samples;
m_uncompressed_offset = 0;
m_uncompressed_length = num_samples;
m_uncompressed_swap = swap_endian;
// loop until we get everything we want
while (m_uncompressed_offset < m_uncompressed_length)
if (!FLAC__stream_decoder_process_single(m_decoder))
return false;
return true;
}
//-------------------------------------------------
// decode - decode to an multiple independent
// data streams
//-------------------------------------------------
bool flac_decoder::decode(INT16 **samples, UINT32 num_samples, bool swap_endian)
{
// make sure we don't have too many channels
int chans = channels();
if (chans > ARRAY_LENGTH(m_uncompressed_start))
return false;
// configure the uncompressed buffer
memset(m_uncompressed_start, 0, sizeof(m_uncompressed_start));
for (int curchan = 0; curchan < chans; curchan++)
m_uncompressed_start[curchan] = samples[curchan];
m_uncompressed_offset = 0;
m_uncompressed_length = num_samples;
m_uncompressed_swap = swap_endian;
// loop until we get everything we want
while (m_uncompressed_offset < m_uncompressed_length)
if (!FLAC__stream_decoder_process_single(m_decoder))
return false;
return true;
}
//-------------------------------------------------
// finish - finish up the decode
//-------------------------------------------------
void flac_decoder::finish()
{
FLAC__stream_decoder_finish(m_decoder);
}
//-------------------------------------------------
// read_callback - handle reads from the input
// stream
//-------------------------------------------------
FLAC__StreamDecoderReadStatus flac_decoder::read_callback_static(const FLAC__StreamDecoder *decoder, FLAC__byte buffer[], size_t *bytes, void *client_data)
{
return reinterpret_cast<flac_decoder *>(client_data)->read_callback(buffer, bytes);
}
FLAC__StreamDecoderReadStatus flac_decoder::read_callback(FLAC__byte buffer[], size_t *bytes)
{
UINT32 expected = *bytes;
// if a file, just read
if (m_file != NULL)
*bytes = core_fread(m_file, buffer, expected);
// otherwise, copy from memory
else
{
// copy from primary buffer first
UINT32 outputpos = 0;
if (outputpos < *bytes && m_compressed_offset < m_compressed_length)
{
UINT32 bytes_to_copy = MIN(*bytes - outputpos, m_compressed_length - m_compressed_offset);
memcpy(&buffer[outputpos], m_compressed_start + m_compressed_offset, bytes_to_copy);
outputpos += bytes_to_copy;
m_compressed_offset += bytes_to_copy;
}
// once we're out of that, copy from the secondary buffer
if (outputpos < *bytes && m_compressed_offset < m_compressed_length + m_compressed2_length)
{
UINT32 bytes_to_copy = MIN(*bytes - outputpos, m_compressed2_length - (m_compressed_offset - m_compressed_length));
memcpy(&buffer[outputpos], m_compressed2_start + m_compressed_offset - m_compressed_length, bytes_to_copy);
outputpos += bytes_to_copy;
m_compressed_offset += bytes_to_copy;
}
*bytes = outputpos;
}
// return based on whether we ran out of data
return (*bytes < expected) ? FLAC__STREAM_DECODER_READ_STATUS_END_OF_STREAM : FLAC__STREAM_DECODER_READ_STATUS_CONTINUE;
}
//-------------------------------------------------
// write_callback - handle writes to the output
// stream
//-------------------------------------------------
FLAC__StreamDecoderWriteStatus flac_decoder::write_callback_static(const FLAC__StreamDecoder *decoder, const ::FLAC__Frame *frame, const FLAC__int32 * const buffer[], void *client_data)
{
return reinterpret_cast<flac_decoder *>(client_data)->write_callback(frame, buffer);
}
FLAC__StreamDecoderWriteStatus flac_decoder::write_callback(const ::FLAC__Frame *frame, const FLAC__int32 * const buffer[])
{
assert(frame->header.channels == channels());
// interleaved case
int shift = m_uncompressed_swap ? 8 : 0;
int blocksize = frame->header.blocksize;
if (m_uncompressed_start[1] == NULL)
{
INT16 *dest = m_uncompressed_start[0] + m_uncompressed_offset * frame->header.channels;
for (int sampnum = 0; sampnum < blocksize && m_uncompressed_offset < m_uncompressed_length; sampnum++, m_uncompressed_offset++)
for (int chan = 0; chan < frame->header.channels; chan++)
*dest++ = INT16((UINT16(buffer[chan][sampnum]) << shift) | (UINT16(buffer[chan][sampnum]) >> shift));
}
// non-interleaved case
else
{
for (int sampnum = 0; sampnum < blocksize && m_uncompressed_offset < m_uncompressed_length; sampnum++, m_uncompressed_offset++)
for (int chan = 0; chan < frame->header.channels; chan++)
if (m_uncompressed_start[chan] != NULL)
m_uncompressed_start[chan][m_uncompressed_offset] = INT16((UINT16(buffer[chan][sampnum]) << shift) | (UINT16(buffer[chan][sampnum]) >> shift));
}
return FLAC__STREAM_DECODER_WRITE_STATUS_CONTINUE;
}
//-------------------------------------------------
// error_callback - handle errors (ignore them)
//-------------------------------------------------
void flac_decoder::error_callback_static(const FLAC__StreamDecoder *decoder, FLAC__StreamDecoderErrorStatus status, void *client_data)
{
}